[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:44:04 dist-test-slave-hvgn krb5kdc[9887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:44:04 dist-test-slave-hvgn krb5kdc[9887](info): set up 2 sockets
Jun 21 02:44:04 dist-test-slave-hvgn krb5kdc[9887](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260621 02:44:05.468438 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.006s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:44:05 dist-test-slave-hvgn krb5kdc[9887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:44:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:44:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:44:05.643709 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:40897
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:46223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:40897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:05.959197 9900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:05.959573 9900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:05.959753 9900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:05.966912 9900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:44:05.967011 9900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:05.967115 9900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:05.967217 9900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:44:05.967304 9900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:44:05.977028 9900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:46223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:40897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:40897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:05.978300 9900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:05.979700 9900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:05.991765 9906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:05.991679 9905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:05.991850 9908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:05.992851 9900 server_base.cc:1034] running on GCE node
I20260621 02:44:05.993571 9900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:05.994891 9900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:05.996136 9900 hybrid_clock.cc:648] HybridClock initialized: now 1782009845996091 us; error 40 us; skew 500 ppm
I20260621 02:44:05.996524 9900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:05 dist-test-slave-hvgn krb5kdc[9887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009845, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:05.999715 9900 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:44:06.001081 9900 webserver.cc:460] Webserver started at http://127.9.166.62:37199/ using document root <none> and password file <none>
I20260621 02:44:06.001693 9900 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:06.001787 9900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:06.002027 9900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:06.004884 9900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "a15bb0242993440a83c59a59123cc518"
format_stamp: "Formatted at 2026-06-21 02:44:05 on dist-test-slave-hvgn"
server_key: "16b315106db90292d28c7ff623fe7c98"
server_key_iv: "9a29e15e186d34875c2e348fcca4fa96"
server_key_version: "encryptionkey@0"
I20260621 02:44:06.005573 9900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "a15bb0242993440a83c59a59123cc518"
format_stamp: "Formatted at 2026-06-21 02:44:05 on dist-test-slave-hvgn"
server_key: "16b315106db90292d28c7ff623fe7c98"
server_key_iv: "9a29e15e186d34875c2e348fcca4fa96"
server_key_version: "encryptionkey@0"
I20260621 02:44:06.010424 9900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260621 02:44:06.013680 9915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:06.014948 9900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 02:44:06.015076 9900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "a15bb0242993440a83c59a59123cc518"
format_stamp: "Formatted at 2026-06-21 02:44:05 on dist-test-slave-hvgn"
server_key: "16b315106db90292d28c7ff623fe7c98"
server_key_iv: "9a29e15e186d34875c2e348fcca4fa96"
server_key_version: "encryptionkey@0"
I20260621 02:44:06.015220 9900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:06.033893 9900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:06.037989 9900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:06.038220 9900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:06.056217 9900 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:40897
I20260621 02:44:06.056252 9966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:40897 every 8 connection(s)
I20260621 02:44:06.057512 9900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:44:06.059849 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 9900
I20260621 02:44:06.060195 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:44:06.062129 9967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:06.062669 9880 external_mini_cluster.cc:1428] Setting key 3c993f3a479328b8f8a655dc09d456b2
I20260621 02:44:06.076514 9967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518: Bootstrap starting.
I20260621 02:44:06.080299 9967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:06.081418 9967 log.cc:824] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:06.085225 9967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518: No bootstrap required, opened a new log
I20260621 02:44:06.095331 9967 raft_consensus.cc:348] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15bb0242993440a83c59a59123cc518" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40897 } }
I20260621 02:44:06.095835 9967 raft_consensus.cc:374] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:06.095959 9967 raft_consensus.cc:729] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15bb0242993440a83c59a59123cc518, State: Initialized, Role: FOLLOWER
Jun 21 02:44:06 dist-test-slave-hvgn krb5kdc[9887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:06.096625 9967 consensus_queue.cc:260] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15bb0242993440a83c59a59123cc518" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40897 } }
I20260621 02:44:06.096855 9967 raft_consensus.cc:388] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:44:06.096983 9967 raft_consensus.cc:482] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:44:06.097154 9967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:06.099545 9967 raft_consensus.cc:504] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15bb0242993440a83c59a59123cc518" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40897 } }
I20260621 02:44:06.100101 9967 leader_election.cc:302] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a15bb0242993440a83c59a59123cc518; no voters:
I20260621 02:44:06.100736 9967 leader_election.cc:288] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:44:06.100920 9972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:06.102159 9972 raft_consensus.cc:686] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518 [term 1 LEADER]: Becoming Leader. State: Replica: a15bb0242993440a83c59a59123cc518, State: Running, Role: LEADER
I20260621 02:44:06.102759 9972 consensus_queue.cc:237] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15bb0242993440a83c59a59123cc518" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40897 } }
I20260621 02:44:06.103147 9967 sys_catalog.cc:564] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:44:06.110083 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:06.067318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33248 (local address 127.9.166.62:40897)
0621 02:44:06.068306 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:06.068325 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:06.069499 (+ 1174us) server_negotiation.cc:407] Connection header received
0621 02:44:06.081281 (+ 11782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:06.081309 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:06.081684 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:06.082062 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:06.086025 (+ 3963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:06.087366 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:06.089799 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:06.090492 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:06.090637 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:06.097698 (+ 7061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:06.097710 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:06.097725 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:06.097770 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:06.101499 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:06.102671 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:06.102680 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:06.102687 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:06.102977 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:06.103580 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:06.103586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:06.103589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:06.104320 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:06.104648 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:06.105955 (+ 1307us) server_negotiation.cc:299] Negotiation successful
0621 02:44:06.106496 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":361,"threads_started":1}
I20260621 02:44:06.110462 9973 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a15bb0242993440a83c59a59123cc518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15bb0242993440a83c59a59123cc518" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40897 } } }
I20260621 02:44:06.110486 9974 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a15bb0242993440a83c59a59123cc518. Latest consensus state: current_term: 1 leader_uuid: "a15bb0242993440a83c59a59123cc518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15bb0242993440a83c59a59123cc518" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40897 } } }
I20260621 02:44:06.110980 9974 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:06.110980 9973 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:06.112643 9982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:44:06.120211 9982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:44:06.129185 9982 catalog_manager.cc:1347] Generated new cluster ID: b3d625fe4b284d23a0cd46db07b6869f
I20260621 02:44:06.129312 9982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:44:06.138417 9982 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:44:06.139642 9982 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:44:06.151599 9982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a15bb0242993440a83c59a59123cc518: Generated new TSK 0
I20260621 02:44:06.152459 9982 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
I20260621 02:44:06.207902 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:40897
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:46223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:06.524472 9993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:06.524796 9993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:06.524976 9993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:06.532328 9993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:06.532478 9993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:06.532672 9993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:44:06.542522 9993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:46223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:40897
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:06.543880 9993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:06.545188 9993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:06.557914 9998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:06.557878 10001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:06.557857 9999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:06.558920 9993 server_base.cc:1034] running on GCE node
I20260621 02:44:06.559594 9993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:06.560603 9993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:06.561849 9993 hybrid_clock.cc:648] HybridClock initialized: now 1782009846561785 us; error 57 us; skew 500 ppm
I20260621 02:44:06.562160 9993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:06 dist-test-slave-hvgn krb5kdc[9887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009846, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:06.565276 9993 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:44:06.566696 9993 webserver.cc:460] Webserver started at http://127.9.166.1:35555/ using document root <none> and password file <none>
I20260621 02:44:06.567461 9993 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:06.567601 9993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:06.567912 9993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:06.570675 9993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "2661429175cf44e2880874646ad7978a"
format_stamp: "Formatted at 2026-06-21 02:44:06 on dist-test-slave-hvgn"
server_key: "bd8eba39218106664a469d5a8e0c051e"
server_key_iv: "2e8216377e3170ffd3293e3aeaefffc7"
server_key_version: "encryptionkey@0"
I20260621 02:44:06.571766 9993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "2661429175cf44e2880874646ad7978a"
format_stamp: "Formatted at 2026-06-21 02:44:06 on dist-test-slave-hvgn"
server_key: "bd8eba39218106664a469d5a8e0c051e"
server_key_iv: "2e8216377e3170ffd3293e3aeaefffc7"
server_key_version: "encryptionkey@0"
I20260621 02:44:06.576666 9993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260621 02:44:06.579895 10008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:06.581212 9993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260621 02:44:06.581403 9993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "2661429175cf44e2880874646ad7978a"
format_stamp: "Formatted at 2026-06-21 02:44:06 on dist-test-slave-hvgn"
server_key: "bd8eba39218106664a469d5a8e0c051e"
server_key_iv: "2e8216377e3170ffd3293e3aeaefffc7"
server_key_version: "encryptionkey@0"
I20260621 02:44:06.581597 9993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:06.628834 9993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:06.633313 9993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:06.633617 9993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:06.634690 9993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:06.636111 9993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:06.636221 9993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:06.636386 9993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:06.636468 9993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:06.671520 9993 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:39241
I20260621 02:44:06.671543 10120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:39241 every 8 connection(s)
I20260621 02:44:06.672897 9993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:44:06.674746 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 9993
I20260621 02:44:06.675010 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:44:06.675926 9880 external_mini_cluster.cc:1428] Setting key 97a490130bab2c4c606cb770a4262f34
Jun 21 02:44:06 dist-test-slave-hvgn krb5kdc[9887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009846, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.2@KRBTEST.COM" created.
I20260621 02:44:06.697059 10123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:06.675606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:35667 (local address 127.9.166.62:40897)
0621 02:44:06.676084 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:06.676089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:06.677429 (+ 1340us) server_negotiation.cc:407] Connection header received
0621 02:44:06.679498 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:06.679507 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:06.679622 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:06.679761 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:06.681234 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:06.682547 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:06.685237 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:06.685772 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:06.686011 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:06.690539 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:06.690548 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:06.690556 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:06.690602 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:06.694169 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:06.694869 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:06.694873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:06.694875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:06.694988 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:06.695341 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:06.695345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:06.695346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:06.695647 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:06.695757 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:06.696479 (+ 722us) server_negotiation.cc:299] Negotiation successful
0621 02:44:06.696694 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":239,"threads_started":1}
I20260621 02:44:06.698033 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:06.676352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:40897 (local address 127.9.166.1:35667)
0621 02:44:06.677244 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:06.677295 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:06.679196 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:06.679982 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:06.679990 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:06.680480 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:06.681038 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:06.681055 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:06.682718 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:06.682724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:06.685094 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:06.685112 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:06.686040 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:06.686049 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:06.686383 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:06.687152 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:06.687187 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:06.690302 (+ 3115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:06.694315 (+ 4013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:06.694321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:06.694335 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:06.694740 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:06.695081 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:06.695084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:06.695086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:06.695256 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:06.695752 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:06.695758 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:06.696099 (+ 341us) client_negotiation.cc:769] Sending connection context
0621 02:44:06.696374 (+ 275us) client_negotiation.cc:240] Negotiation successful
0621 02:44:06.696900 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":397,"threads_started":1}
I20260621 02:44:06.699620 10121 heartbeater.cc:347] Connected to a master server at 127.9.166.62:40897
I20260621 02:44:06.699971 10121 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:06.700657 10121 heartbeater.cc:511] Master 127.9.166.62:40897 requested a full tablet report, sending...
I20260621 02:44:06.702890 9932 ts_manager.cc:194] Registered new tserver with Master: 2661429175cf44e2880874646ad7978a (127.9.166.1:39241)
I20260621 02:44:06.705137 9932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:35667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
I20260621 02:44:06.716657 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:40897
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:46223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:07.031351 10127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:07.031633 10127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:07.031729 10127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:07.038910 10127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:07.039021 10127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:07.039134 10127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:44:07.048650 10127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:46223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:40897
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:07.049871 10127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:07.051225 10127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:07.064468 10133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:07.064502 10135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:07.065008 10132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:07.065471 10127 server_base.cc:1034] running on GCE node
I20260621 02:44:07.066092 10127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:07.067130 10127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:07.068356 10127 hybrid_clock.cc:648] HybridClock initialized: now 1782009847068278 us; error 67 us; skew 500 ppm
I20260621 02:44:07.068699 10127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:07 dist-test-slave-hvgn krb5kdc[9887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:07.072321 10127 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:44:07.073770 10127 webserver.cc:460] Webserver started at http://127.9.166.2:40277/ using document root <none> and password file <none>
I20260621 02:44:07.074456 10127 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:07.074581 10127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:07.074903 10127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:07.077816 10127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "d348a1d6172649a5bf662dc5d1686d5c"
format_stamp: "Formatted at 2026-06-21 02:44:07 on dist-test-slave-hvgn"
server_key: "065893e5c08c5779f99c35e3c61db777"
server_key_iv: "bc1e1d185ee79a0137e6fa280a34e261"
server_key_version: "encryptionkey@0"
I20260621 02:44:07.078562 10127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "d348a1d6172649a5bf662dc5d1686d5c"
format_stamp: "Formatted at 2026-06-21 02:44:07 on dist-test-slave-hvgn"
server_key: "065893e5c08c5779f99c35e3c61db777"
server_key_iv: "bc1e1d185ee79a0137e6fa280a34e261"
server_key_version: "encryptionkey@0"
I20260621 02:44:07.083714 10127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260621 02:44:07.087117 10142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:07.088447 10127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:44:07.088616 10127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "d348a1d6172649a5bf662dc5d1686d5c"
format_stamp: "Formatted at 2026-06-21 02:44:07 on dist-test-slave-hvgn"
server_key: "065893e5c08c5779f99c35e3c61db777"
server_key_iv: "bc1e1d185ee79a0137e6fa280a34e261"
server_key_version: "encryptionkey@0"
I20260621 02:44:07.088816 10127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:07.115417 10127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:07.119992 10127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:07.120287 10127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:07.121299 10127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:07.122845 10127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:07.122941 10127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:07.123091 10127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:07.123159 10127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:07.155850 10127 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:39951
I20260621 02:44:07.155896 10254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:39951 every 8 connection(s)
I20260621 02:44:07.157083 10127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:44:07.162194 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 10127
I20260621 02:44:07.162395 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:44:07.162933 9880 external_mini_cluster.cc:1428] Setting key 2c72b9cfeaa67d53d3b61fc9ec379d5d
Jun 21 02:44:07 dist-test-slave-hvgn krb5kdc[9887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:07.178884 10123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:07.159559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:56809 (local address 127.9.166.62:40897)
0621 02:44:07.159726 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:07.159731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:07.160851 (+ 1120us) server_negotiation.cc:407] Connection header received
0621 02:44:07.162664 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:07.162673 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:07.162757 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:07.162866 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:07.164493 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:07.165306 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:07.166821 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:07.167250 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:07.167383 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:07.171483 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:07.171493 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:07.171502 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:07.171551 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:07.175565 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:07.176248 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:07.176257 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:07.176261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:07.176432 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:07.176858 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:07.176865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:07.176871 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:07.177297 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:07.177519 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:07.178116 (+ 597us) server_negotiation.cc:299] Negotiation successful
0621 02:44:07.178389 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:44:07.179412 10257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:07.160015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:40897 (local address 127.9.166.2:56809)
0621 02:44:07.160685 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:07.160721 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:07.162348 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:07.163094 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:07.163101 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:07.163749 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:07.164314 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:07.164332 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:07.165457 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:07.165466 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:07.166675 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:07.166690 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:07.167486 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:07.167493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:07.167717 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:07.168526 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:07.168565 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:07.171214 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:07.175692 (+ 4478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:07.175699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:07.175710 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:07.176106 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:07.176560 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:07.176563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:07.176565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:07.176724 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:07.177429 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:07.177435 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:07.177734 (+ 299us) client_negotiation.cc:769] Sending connection context
0621 02:44:07.177945 (+ 211us) client_negotiation.cc:240] Negotiation successful
0621 02:44:07.178296 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":269,"threads_started":1}
WARNING: no policy specified for kudu/127.9.166.3@KRBTEST.COM; defaulting to no policy
I20260621 02:44:07.180967 10255 heartbeater.cc:347] Connected to a master server at 127.9.166.62:40897
I20260621 02:44:07.181305 10255 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:07.182004 10255 heartbeater.cc:511] Master 127.9.166.62:40897 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.3@KRBTEST.COM" created.
I20260621 02:44:07.183511 9932 ts_manager.cc:194] Registered new tserver with Master: d348a1d6172649a5bf662dc5d1686d5c (127.9.166.2:39951)
I20260621 02:44:07.184309 9932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:56809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
I20260621 02:44:07.203205 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:40897
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:46223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:07.530094 10260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:07.530366 10260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:07.530460 10260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:07.537781 10260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:07.537883 10260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:07.537997 10260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:44:07.547369 10260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:46223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:40897
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:07.548571 10260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:07.549784 10260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:07.562392 10268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:07.562465 10265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:07.562461 10266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:07.563652 10260 server_base.cc:1034] running on GCE node
I20260621 02:44:07.564313 10260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:07.565224 10260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:07.566421 10260 hybrid_clock.cc:648] HybridClock initialized: now 1782009847566378 us; error 31 us; skew 500 ppm
I20260621 02:44:07.566728 10260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:07 dist-test-slave-hvgn krb5kdc[9887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:07.569900 10260 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:44:07.571642 10260 webserver.cc:460] Webserver started at http://127.9.166.3:37877/ using document root <none> and password file <none>
I20260621 02:44:07.572326 10260 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:07.572479 10260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:07.572788 10260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:07.575593 10260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "12475b738dc048e7b7b4b8923c914e41"
format_stamp: "Formatted at 2026-06-21 02:44:07 on dist-test-slave-hvgn"
server_key: "65a0dcad0432e923d00611e36986cdbd"
server_key_iv: "7745669753a4c4d9e75f97be8930ed31"
server_key_version: "encryptionkey@0"
I20260621 02:44:07.576364 10260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "12475b738dc048e7b7b4b8923c914e41"
format_stamp: "Formatted at 2026-06-21 02:44:07 on dist-test-slave-hvgn"
server_key: "65a0dcad0432e923d00611e36986cdbd"
server_key_iv: "7745669753a4c4d9e75f97be8930ed31"
server_key_version: "encryptionkey@0"
I20260621 02:44:07.581125 10260 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260621 02:44:07.584403 10275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:07.585777 10260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:44:07.585942 10260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "12475b738dc048e7b7b4b8923c914e41"
format_stamp: "Formatted at 2026-06-21 02:44:07 on dist-test-slave-hvgn"
server_key: "65a0dcad0432e923d00611e36986cdbd"
server_key_iv: "7745669753a4c4d9e75f97be8930ed31"
server_key_version: "encryptionkey@0"
I20260621 02:44:07.586160 10260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:07.606779 10260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:07.611155 10260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:07.611511 10260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:07.612375 10260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:07.613541 10260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:07.613613 10260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:07.613696 10260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:07.613740 10260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:07.646063 10260 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:46407
I20260621 02:44:07.646116 10387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:46407 every 8 connection(s)
I20260621 02:44:07.647369 10260 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:44:07.647847 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 10260
I20260621 02:44:07.648037 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:44:07.648636 9880 external_mini_cluster.cc:1428] Setting key 4f8af6872e18c309fa2c3bc943ace797
Jun 21 02:44:07 dist-test-slave-hvgn krb5kdc[9887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:07.666829 10123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:07.650015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:47643 (local address 127.9.166.62:40897)
0621 02:44:07.650227 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:07.650233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:07.651660 (+ 1427us) server_negotiation.cc:407] Connection header received
0621 02:44:07.653641 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:07.653646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:07.653742 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:07.653844 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:07.655237 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:07.656062 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:07.657537 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:07.657942 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:07.658070 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:07.661407 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:07.661414 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:07.661419 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:07.661458 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:07.664234 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:07.664985 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:07.664989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:07.664991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:07.665126 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:07.665520 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:07.665524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:07.665526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:07.665785 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:07.665891 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:07.666444 (+ 553us) server_negotiation.cc:299] Negotiation successful
0621 02:44:07.666584 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:44:07.667809 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:07.650742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:40897 (local address 127.9.166.3:47643)
0621 02:44:07.651528 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:07.651564 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:07.653400 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:07.654066 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:07.654073 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:07.654564 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:07.655083 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:07.655098 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:07.656198 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:07.656209 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:07.657397 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:07.657408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:07.658159 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:07.658166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:07.658405 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:07.659101 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:07.659126 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:07.661207 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:07.664352 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:07.664359 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:07.664370 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:07.664859 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:07.665227 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:07.665230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:07.665232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:07.665411 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:07.665883 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:07.665888 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:07.666150 (+ 262us) client_negotiation.cc:769] Sending connection context
0621 02:44:07.666360 (+ 210us) client_negotiation.cc:240] Negotiation successful
0621 02:44:07.666690 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":321,"threads_started":1}
I20260621 02:44:07.669036 10388 heartbeater.cc:347] Connected to a master server at 127.9.166.62:40897
I20260621 02:44:07.669328 10388 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:07.669982 10388 heartbeater.cc:511] Master 127.9.166.62:40897 requested a full tablet report, sending...
I20260621 02:44:07.671443 9932 ts_manager.cc:194] Registered new tserver with Master: 12475b738dc048e7b7b4b8923c914e41 (127.9.166.3:46407)
I20260621 02:44:07.672549 9932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:47643
I20260621 02:44:07.676739 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 02:44:07 dist-test-slave-hvgn krb5kdc[9887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009847, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260621 02:44:07.708024 10121 heartbeater.cc:503] Master 127.9.166.62:40897 was elected leader, sending a full tablet report...
Jun 21 02:44:07 dist-test-slave-hvgn krb5kdc[9887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009847, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:44:07.719937 10395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:07.705020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40938 (local address 127.9.166.1:39241)
0621 02:44:07.705676 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:07.705684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:07.705712 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:44:07.705885 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:07.705892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:07.706116 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:07.706264 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:07.707678 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:07.708932 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:07.710394 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:07.710938 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:07.711028 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:07.714751 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:07.714762 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:07.714776 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:07.714805 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:07.717444 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:07.718018 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:07.718028 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:07.718032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:07.718154 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:07.718483 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:07.718486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:07.718488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:07.718850 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:07.719066 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:07.719301 (+ 235us) server_negotiation.cc:299] Negotiation successful
0621 02:44:07.719556 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":371,"threads_started":1}
W20260621 02:44:07.721131 10036 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40938
Jun 21 02:44:07 dist-test-slave-hvgn krb5kdc[9887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 02:44:07 dist-test-slave-hvgn krb5kdc[9887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:44:07.762611 10395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:07.749387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40948 (local address 127.9.166.1:39241)
0621 02:44:07.749607 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:07.749617 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:07.749917 (+ 300us) server_negotiation.cc:407] Connection header received
0621 02:44:07.750129 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:07.750133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:07.750206 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:07.750333 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:07.751653 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:07.752344 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:07.753553 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:07.753956 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:07.754078 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:07.757829 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:07.757835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:07.757838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:07.757869 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:07.760222 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:07.760826 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:07.760834 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:07.760837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:07.760957 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:07.761312 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:07.761317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:07.761321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:07.761553 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:07.761782 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:07.762087 (+ 305us) server_negotiation.cc:299] Negotiation successful
0621 02:44:07.762316 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:44:07.763986 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 9993
I20260621 02:44:07.775219 10395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:07.764834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40962 (local address 127.9.166.1:39241)
0621 02:44:07.765016 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:07.765023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:07.765320 (+ 297us) server_negotiation.cc:407] Connection header received
0621 02:44:07.765409 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:07.765413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:07.765463 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:07.765533 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:07.766578 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:07.767198 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:07.768440 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:07.768845 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:07.768931 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:07.770922 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:07.770926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:07.770929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:07.770953 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:07.773154 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:07.773714 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:07.773717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:07.773719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:07.773833 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:07.774113 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:07.774116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:07.774118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:07.774352 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:07.774513 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:07.774739 (+ 226us) server_negotiation.cc:299] Negotiation successful
0621 02:44:07.774874 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 02:44:07.775909 10116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:07.858057 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 9993
I20260621 02:44:07.873857 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 10127
Jun 21 02:44:07 dist-test-slave-hvgn krb5kdc[9887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.2@KRBTEST.COM
I20260621 02:44:07.888638 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:07.874788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.9.166.2:39951)
0621 02:44:07.875368 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:07.875375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:07.875446 (+ 71us) server_negotiation.cc:407] Connection header received
0621 02:44:07.875509 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:07.875516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:07.875726 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:07.875861 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:07.876962 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:07.877941 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:07.879094 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:07.879710 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:07.879819 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:07.883597 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:07.883608 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:07.883621 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:07.883652 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:07.886197 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:07.886810 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:07.886817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:07.886819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:07.886934 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:07.887280 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:07.887283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:07.887285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:07.887694 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:07.887913 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:07.888153 (+ 240us) server_negotiation.cc:299] Negotiation successful
0621 02:44:07.888349 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":300,"threads_started":1}
I20260621 02:44:07.890053 10250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:07.972106 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 10127
I20260621 02:44:07.987708 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 10260
Jun 21 02:44:07 dist-test-slave-hvgn krb5kdc[9887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.3@KRBTEST.COM
I20260621 02:44:08.002792 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:07.988578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46330 (local address 127.9.166.3:46407)
0621 02:44:07.989238 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:07.989245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:07.989274 (+ 29us) server_negotiation.cc:407] Connection header received
0621 02:44:07.989363 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:07.989372 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:07.989655 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:07.989814 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:07.991015 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:07.992096 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:07.993327 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:07.993844 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:07.993942 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:07.997873 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:07.997884 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:07.997896 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:07.997925 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:08.000494 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:08.001051 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:08.001057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:08.001059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:08.001167 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:08.001492 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:08.001495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:08.001497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:08.001798 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:08.002026 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:08.002315 (+ 289us) server_negotiation.cc:299] Negotiation successful
0621 02:44:08.002499 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":326,"threads_started":1}
I20260621 02:44:08.004136 10383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:08.093509 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 10260
I20260621 02:44:08.109371 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 9900
I20260621 02:44:08.110737 9962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:08.210721 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 9900
2026-06-21T02:44:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3825 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:44:08 dist-test-slave-hvgn krb5kdc[10409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:44:08 dist-test-slave-hvgn krb5kdc[10409](info): set up 2 sockets
Jun 21 02:44:08 dist-test-slave-hvgn krb5kdc[10409](info): commencing operation
krb5kdc: starting...
W20260621 02:44:09.299191 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.006s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:44:09 dist-test-slave-hvgn krb5kdc[10409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:44:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:44:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:44:09.464284 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:34135
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:34135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:09.774286 10422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:09.774650 10422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:09.774824 10422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:09.782553 10422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:44:09.782735 10422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:09.782851 10422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:09.782968 10422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:44:09.783048 10422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:44:09.793399 10422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:34135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:34135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:09.794910 10422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:09.796386 10422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:09.808917 10427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:09.809255 10428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:09.809350 10430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:09.810220 10422 server_base.cc:1034] running on GCE node
I20260621 02:44:09.811067 10422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:09.812516 10422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:09.813730 10422 hybrid_clock.cc:648] HybridClock initialized: now 1782009849813678 us; error 36 us; skew 500 ppm
I20260621 02:44:09.814097 10422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:09 dist-test-slave-hvgn krb5kdc[10409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009849, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:09.820463 10422 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:44:09.821934 10422 webserver.cc:460] Webserver started at http://127.9.166.62:40761/ using document root <none> and password file <none>
I20260621 02:44:09.822582 10422 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:09.822678 10422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:09.822929 10422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:09.825839 10422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "142eece39c524a0c9874d93e59d25e58"
format_stamp: "Formatted at 2026-06-21 02:44:09 on dist-test-slave-hvgn"
server_key: "fc76a5304585699e8a8c6fb344f57ae8"
server_key_iv: "63ddc25d87a6f4a5b3bcf3d171434171"
server_key_version: "encryptionkey@0"
I20260621 02:44:09.826524 10422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "142eece39c524a0c9874d93e59d25e58"
format_stamp: "Formatted at 2026-06-21 02:44:09 on dist-test-slave-hvgn"
server_key: "fc76a5304585699e8a8c6fb344f57ae8"
server_key_iv: "63ddc25d87a6f4a5b3bcf3d171434171"
server_key_version: "encryptionkey@0"
I20260621 02:44:09.831677 10422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 02:44:09.834986 10437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:09.836383 10422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:44:09.836509 10422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "142eece39c524a0c9874d93e59d25e58"
format_stamp: "Formatted at 2026-06-21 02:44:09 on dist-test-slave-hvgn"
server_key: "fc76a5304585699e8a8c6fb344f57ae8"
server_key_iv: "63ddc25d87a6f4a5b3bcf3d171434171"
server_key_version: "encryptionkey@0"
I20260621 02:44:09.836660 10422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:09.868917 10422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:09.873226 10422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:09.873453 10422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:09.891508 10422 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:34135
I20260621 02:44:09.891599 10488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:34135 every 8 connection(s)
I20260621 02:44:09.892822 10422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:44:09.896399 10489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:09.899099 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 10422
I20260621 02:44:09.899313 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:44:09.900163 9880 external_mini_cluster.cc:1428] Setting key d65c8f1a6faf43b4a0a645996edf50c2
I20260621 02:44:09.908562 10489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58: Bootstrap starting.
I20260621 02:44:09.912186 10489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:09.913478 10489 log.cc:824] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58: Log is configured to *not* fsync() on all Append() calls
Jun 21 02:44:09 dist-test-slave-hvgn krb5kdc[10409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:09.916426 10489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58: No bootstrap required, opened a new log
I20260621 02:44:09.922887 10492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:09.902193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.9.166.62:34135)
0621 02:44:09.902912 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:09.902922 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:09.902960 (+ 38us) server_negotiation.cc:407] Connection header received
0621 02:44:09.904580 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:09.904602 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:09.904973 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:09.905285 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:09.907031 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:09.908351 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:09.910142 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:09.910772 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:09.910903 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:09.915777 (+ 4874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:09.915785 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:09.915798 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:09.915830 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:09.918939 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:09.919695 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:09.919702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:09.919711 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:09.919868 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:09.920235 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:09.920240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:09.920243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:09.920733 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:09.921072 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:09.921559 (+ 487us) server_negotiation.cc:299] Negotiation successful
0621 02:44:09.921939 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":338,"threads_started":1}
I20260621 02:44:09.924396 10489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142eece39c524a0c9874d93e59d25e58" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34135 } }
I20260621 02:44:09.924750 10489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:09.924854 10489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142eece39c524a0c9874d93e59d25e58, State: Initialized, Role: FOLLOWER
I20260621 02:44:09.925415 10489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142eece39c524a0c9874d93e59d25e58" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34135 } }
I20260621 02:44:09.925673 10489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:44:09.925810 10489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:44:09.925968 10489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:09.928030 10489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142eece39c524a0c9874d93e59d25e58" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34135 } }
I20260621 02:44:09.928428 10489 leader_election.cc:302] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 142eece39c524a0c9874d93e59d25e58; no voters:
I20260621 02:44:09.928951 10489 leader_election.cc:288] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:44:09.929105 10494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:09.930372 10494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58 [term 1 LEADER]: Becoming Leader. State: Replica: 142eece39c524a0c9874d93e59d25e58, State: Running, Role: LEADER
I20260621 02:44:09.930856 10494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142eece39c524a0c9874d93e59d25e58" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34135 } }
I20260621 02:44:09.931031 10489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:44:09.934054 10496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 142eece39c524a0c9874d93e59d25e58. Latest consensus state: current_term: 1 leader_uuid: "142eece39c524a0c9874d93e59d25e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142eece39c524a0c9874d93e59d25e58" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34135 } } }
I20260621 02:44:09.934021 10495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "142eece39c524a0c9874d93e59d25e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142eece39c524a0c9874d93e59d25e58" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34135 } } }
I20260621 02:44:09.934405 10496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:09.934405 10495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:09.935197 10502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:44:09.939944 10502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:44:09.948429 10502 catalog_manager.cc:1347] Generated new cluster ID: 70e12c7cc8bd4d398e090e4d140fd258
I20260621 02:44:09.948738 10502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:44:09.971861 10502 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:44:09.973160 10502 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:44:09.980263 10502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 142eece39c524a0c9874d93e59d25e58: Generated new TSK 0
I20260621 02:44:09.981019 10502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
I20260621 02:44:10.026580 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34135
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:10.367003 10515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:10.367268 10515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:10.367480 10515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:10.374645 10515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:10.374739 10515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:10.374940 10515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:44:10.384469 10515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34135
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:10.385661 10515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:10.386938 10515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:10.399582 10520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:10.399538 10521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:10.399622 10523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:10.400314 10515 server_base.cc:1034] running on GCE node
I20260621 02:44:10.400906 10515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:10.402146 10515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:10.403368 10515 hybrid_clock.cc:648] HybridClock initialized: now 1782009850403309 us; error 47 us; skew 500 ppm
I20260621 02:44:10.403772 10515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:10 dist-test-slave-hvgn krb5kdc[10409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009850, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:10.407248 10515 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:44:10.408589 10515 webserver.cc:460] Webserver started at http://127.9.166.1:42695/ using document root <none> and password file <none>
I20260621 02:44:10.409253 10515 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:10.409390 10515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:10.409690 10515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:10.412498 10515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "df26c5f8d4db4278b0d74abfdd69a570"
format_stamp: "Formatted at 2026-06-21 02:44:10 on dist-test-slave-hvgn"
server_key: "ace7b7fadac0fad6359479973ef42a6b"
server_key_iv: "0f730c185a2a99d59a815a86e45eb0a3"
server_key_version: "encryptionkey@0"
I20260621 02:44:10.413290 10515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "df26c5f8d4db4278b0d74abfdd69a570"
format_stamp: "Formatted at 2026-06-21 02:44:10 on dist-test-slave-hvgn"
server_key: "ace7b7fadac0fad6359479973ef42a6b"
server_key_iv: "0f730c185a2a99d59a815a86e45eb0a3"
server_key_version: "encryptionkey@0"
I20260621 02:44:10.418228 10515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 02:44:10.421567 10530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:10.422971 10515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 02:44:10.423094 10515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "df26c5f8d4db4278b0d74abfdd69a570"
format_stamp: "Formatted at 2026-06-21 02:44:10 on dist-test-slave-hvgn"
server_key: "ace7b7fadac0fad6359479973ef42a6b"
server_key_iv: "0f730c185a2a99d59a815a86e45eb0a3"
server_key_version: "encryptionkey@0"
I20260621 02:44:10.423249 10515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:10.445058 10515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:10.449159 10515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:10.449455 10515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:10.450353 10515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:10.451699 10515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:10.451820 10515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:10.451964 10515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:10.452049 10515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:10.484143 10515 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:36201
I20260621 02:44:10.484237 10642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:36201 every 8 connection(s)
I20260621 02:44:10.485422 10515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:44:10.493992 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 10515
I20260621 02:44:10.494201 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:44:10.494809 9880 external_mini_cluster.cc:1428] Setting key 86cd9dd0f0ead0fc1fbe53bd14de0041
Jun 21 02:44:10 dist-test-slave-hvgn krb5kdc[10409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009850, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:10.505632 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:10.487978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:44721 (local address 127.9.166.62:34135)
0621 02:44:10.488555 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:10.488560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:10.489519 (+ 959us) server_negotiation.cc:407] Connection header received
0621 02:44:10.491282 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:10.491286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:10.491349 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:10.491495 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:10.492934 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:10.493724 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:10.495473 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:10.495945 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:10.496081 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:10.499852 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:10.499857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:10.499860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:10.499889 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:10.502805 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:10.503518 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:10.503525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:10.503528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:10.503644 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:10.504049 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:10.504056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:10.504059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:10.504304 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:10.504454 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:10.505086 (+ 632us) server_negotiation.cc:299] Negotiation successful
0621 02:44:10.505297 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":364,"threads_started":1}
I20260621 02:44:10.506556 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:10.488646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34135 (local address 127.9.166.1:44721)
0621 02:44:10.489357 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:10.489398 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:10.491032 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:10.491735 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:10.491744 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:10.492250 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:10.492788 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:10.492805 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:10.493890 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:10.493893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:10.495259 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:10.495272 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:10.496144 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:10.496149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:10.496334 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:10.497051 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:10.497089 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:10.499631 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:10.502946 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:10.502954 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:10.502969 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:10.503363 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:10.503751 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:10.503757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:10.503761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:10.503944 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:10.504430 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:10.504438 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:10.504733 (+ 295us) client_negotiation.cc:769] Sending connection context
0621 02:44:10.504977 (+ 244us) client_negotiation.cc:240] Negotiation successful
0621 02:44:10.505460 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":260,"threads_started":1}
I20260621 02:44:10.508054 10643 heartbeater.cc:347] Connected to a master server at 127.9.166.62:34135
I20260621 02:44:10.508339 10643 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:10.508993 10643 heartbeater.cc:511] Master 127.9.166.62:34135 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.166.2@KRBTEST.COM; defaulting to no policy
I20260621 02:44:10.511226 10454 ts_manager.cc:194] Registered new tserver with Master: df26c5f8d4db4278b0d74abfdd69a570 (127.9.166.1:36201)
I20260621 02:44:10.512699 10454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:44721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
I20260621 02:44:10.534065 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34135
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:10.842935 10649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:10.843186 10649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:10.843351 10649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:10.850376 10649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:10.850483 10649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:10.850687 10649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:44:10.860462 10649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34135
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:10.861661 10649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:10.862947 10649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:10.875509 10655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:10.875486 10654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:10.876042 10657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:10.876490 10649 server_base.cc:1034] running on GCE node
I20260621 02:44:10.877158 10649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:10.878050 10649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:10.879277 10649 hybrid_clock.cc:648] HybridClock initialized: now 1782009850879233 us; error 56 us; skew 500 ppm
I20260621 02:44:10.879621 10649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:10 dist-test-slave-hvgn krb5kdc[10409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009850, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:10.882556 10649 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:44:10.883926 10649 webserver.cc:460] Webserver started at http://127.9.166.2:36069/ using document root <none> and password file <none>
I20260621 02:44:10.884579 10649 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:10.884716 10649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:10.885044 10649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:10.887907 10649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "818b3a86f93348e99800cea085333056"
format_stamp: "Formatted at 2026-06-21 02:44:10 on dist-test-slave-hvgn"
server_key: "83f29a60ceb6778628e43f312da9ed3f"
server_key_iv: "8f626e522ca01a1f548b376f8d5dfbf6"
server_key_version: "encryptionkey@0"
I20260621 02:44:10.888679 10649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "818b3a86f93348e99800cea085333056"
format_stamp: "Formatted at 2026-06-21 02:44:10 on dist-test-slave-hvgn"
server_key: "83f29a60ceb6778628e43f312da9ed3f"
server_key_iv: "8f626e522ca01a1f548b376f8d5dfbf6"
server_key_version: "encryptionkey@0"
I20260621 02:44:10.893817 10649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:44:10.897233 10664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:10.898541 10649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:44:10.898710 10649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "818b3a86f93348e99800cea085333056"
format_stamp: "Formatted at 2026-06-21 02:44:10 on dist-test-slave-hvgn"
server_key: "83f29a60ceb6778628e43f312da9ed3f"
server_key_iv: "8f626e522ca01a1f548b376f8d5dfbf6"
server_key_version: "encryptionkey@0"
I20260621 02:44:10.898890 10649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:10.923372 10649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:10.927570 10649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:10.927850 10649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:10.928826 10649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:10.930119 10649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:10.930219 10649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:10.930367 10649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:10.930446 10649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:10.961563 10649 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:42771
I20260621 02:44:10.961608 10776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:42771 every 8 connection(s)
I20260621 02:44:10.962812 10649 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:44:10.970304 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 10649
I20260621 02:44:10.970527 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:44:10.970949 9880 external_mini_cluster.cc:1428] Setting key a9d8b04ae49c5dac02ce151b0783c715
Jun 21 02:44:10 dist-test-slave-hvgn krb5kdc[10409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009850, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:10.984838 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:10.965450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:57165 (local address 127.9.166.62:34135)
0621 02:44:10.965633 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:10.965637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:10.966719 (+ 1082us) server_negotiation.cc:407] Connection header received
0621 02:44:10.968643 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:10.968649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:10.968751 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:10.968860 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:10.970509 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:10.971338 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:10.973261 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:10.973681 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:10.973775 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:10.977986 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:10.977990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:10.977993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:10.978024 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:10.981548 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:10.982251 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:10.982259 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:10.982265 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:10.982433 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:10.982834 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:10.982842 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:10.982848 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:10.983228 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:10.983381 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:10.984105 (+ 724us) server_negotiation.cc:299] Negotiation successful
0621 02:44:10.984365 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:44:10.985579 10779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:10.965875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34135 (local address 127.9.166.2:57165)
0621 02:44:10.966560 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:10.966600 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:10.968348 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:10.969090 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:10.969098 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:10.969591 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:10.970312 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:10.970337 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:10.971532 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:10.971537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:10.973123 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:10.973143 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:10.973897 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:10.973906 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:10.974255 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:10.974991 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:10.975026 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:10.977753 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:10.981692 (+ 3939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:10.981698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:10.981710 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:10.982130 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:10.982540 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:10.982544 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:10.982546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:10.982723 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:10.983346 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:10.983351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:10.983732 (+ 381us) client_negotiation.cc:769] Sending connection context
0621 02:44:10.983964 (+ 232us) client_negotiation.cc:240] Negotiation successful
0621 02:44:10.984349 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":267,"threads_started":1}
I20260621 02:44:10.987155 10777 heartbeater.cc:347] Connected to a master server at 127.9.166.62:34135
I20260621 02:44:10.987536 10777 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:10.988461 10777 heartbeater.cc:511] Master 127.9.166.62:34135 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.166.3@KRBTEST.COM; defaulting to no policy
I20260621 02:44:10.990099 10454 ts_manager.cc:194] Registered new tserver with Master: 818b3a86f93348e99800cea085333056 (127.9.166.2:42771)
I20260621 02:44:10.990996 10454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:57165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
I20260621 02:44:11.010222 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34135
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:11.326752 10782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:11.326996 10782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:11.327086 10782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:11.334187 10782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:11.334280 10782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:11.334386 10782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:44:11.343899 10782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34135
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:11.344990 10782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:11.346204 10782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:11.358536 10787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:11.358636 10788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:11.358596 10790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:11.359362 10782 server_base.cc:1034] running on GCE node
I20260621 02:44:11.359905 10782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:11.360777 10782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:11.361999 10782 hybrid_clock.cc:648] HybridClock initialized: now 1782009851361966 us; error 53 us; skew 500 ppm
I20260621 02:44:11.362295 10782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:11 dist-test-slave-hvgn krb5kdc[10409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009851, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:11.365442 10782 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:44:11.366724 10782 webserver.cc:460] Webserver started at http://127.9.166.3:38645/ using document root <none> and password file <none>
I20260621 02:44:11.367304 10782 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:11.367452 10782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:11.367731 10782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:11.370440 10782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "6439f40d119742bd996e861f4b4d062c"
format_stamp: "Formatted at 2026-06-21 02:44:11 on dist-test-slave-hvgn"
server_key: "d06d6a4ea7c4a0b105208ff4a98181c9"
server_key_iv: "fe1cec71717ca160f10e7b7edf87903c"
server_key_version: "encryptionkey@0"
I20260621 02:44:11.371125 10782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "6439f40d119742bd996e861f4b4d062c"
format_stamp: "Formatted at 2026-06-21 02:44:11 on dist-test-slave-hvgn"
server_key: "d06d6a4ea7c4a0b105208ff4a98181c9"
server_key_iv: "fe1cec71717ca160f10e7b7edf87903c"
server_key_version: "encryptionkey@0"
I20260621 02:44:11.375795 10782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260621 02:44:11.378991 10797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:11.380306 10782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:44:11.380429 10782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "6439f40d119742bd996e861f4b4d062c"
format_stamp: "Formatted at 2026-06-21 02:44:11 on dist-test-slave-hvgn"
server_key: "d06d6a4ea7c4a0b105208ff4a98181c9"
server_key_iv: "fe1cec71717ca160f10e7b7edf87903c"
server_key_version: "encryptionkey@0"
I20260621 02:44:11.380571 10782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:11.401437 10782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:11.405934 10782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:11.406250 10782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:11.407202 10782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:11.408481 10782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:11.408555 10782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:11.408700 10782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:11.408758 10782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:11.440994 10782 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:41903
I20260621 02:44:11.441134 10909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:41903 every 8 connection(s)
I20260621 02:44:11.442430 10782 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:44:11.445523 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 10782
I20260621 02:44:11.445722 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:44:11.446321 9880 external_mini_cluster.cc:1428] Setting key fa4740648dee8a9b2f0aa5de83ababe3
Jun 21 02:44:11 dist-test-slave-hvgn krb5kdc[10409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009851, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:11.462460 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:11.444998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:49739 (local address 127.9.166.62:34135)
0621 02:44:11.445184 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:11.445189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:11.446693 (+ 1504us) server_negotiation.cc:407] Connection header received
0621 02:44:11.449134 (+ 2441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:11.449139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:11.449251 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:11.449363 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:11.450817 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.451629 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.453065 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.453442 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.453544 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:11.456866 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:11.456879 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:11.456882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:11.456911 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:11.459288 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:11.460074 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:11.460082 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:11.460090 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:11.460204 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:11.460667 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:11.460672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:11.460675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:11.461019 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:11.461177 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:11.461909 (+ 732us) server_negotiation.cc:299] Negotiation successful
0621 02:44:11.462112 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 02:44:11.463359 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:11.445674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34135 (local address 127.9.166.3:49739)
0621 02:44:11.446510 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:11.446557 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:11.448850 (+ 2293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:11.449557 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:11.449567 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:11.450058 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:11.450668 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:11.450685 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.451744 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.451748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:11.452948 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:11.452959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.453621 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.453625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:11.453826 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:11.454545 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:11.454568 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:11.456668 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:11.459442 (+ 2774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:11.459450 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:11.459466 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:11.459958 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:11.460314 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:11.460319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:11.460323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:11.460543 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:11.461154 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:11.461161 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:11.461496 (+ 335us) client_negotiation.cc:769] Sending connection context
0621 02:44:11.461800 (+ 304us) client_negotiation.cc:240] Negotiation successful
0621 02:44:11.462293 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":328,"threads_started":1}
I20260621 02:44:11.464581 10910 heartbeater.cc:347] Connected to a master server at 127.9.166.62:34135
I20260621 02:44:11.464836 10910 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:11.465454 10910 heartbeater.cc:511] Master 127.9.166.62:34135 requested a full tablet report, sending...
I20260621 02:44:11.466755 10454 ts_manager.cc:194] Registered new tserver with Master: 6439f40d119742bd996e861f4b4d062c (127.9.166.3:41903)
I20260621 02:44:11.467540 10454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:49739
I20260621 02:44:11.473340 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:44:11.492672 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:11.479214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50862 (local address 127.9.166.62:34135)
0621 02:44:11.479426 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:11.479434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:11.479958 (+ 524us) server_negotiation.cc:407] Connection header received
0621 02:44:11.480152 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:11.480157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:11.480227 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:11.480324 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:11.481704 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.482405 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.483926 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.484441 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.484570 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:11.486866 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:11.486872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:11.486877 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:11.486916 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:11.490048 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:11.490684 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:11.490691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:11.490695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:11.490826 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:11.491204 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:11.491209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:11.491213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:11.491578 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:11.491827 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:11.492174 (+ 347us) server_negotiation.cc:299] Negotiation successful
0621 02:44:11.492378 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 02:44:11.498337 10454 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:44:11.505180 10454 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:44:11.515832 10643 heartbeater.cc:503] Master 127.9.166.62:34135 was elected leader, sending a full tablet report...
I20260621 02:44:11.529063 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:11.517015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55076 (local address 127.9.166.1:36201)
0621 02:44:11.517675 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:11.517684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:11.518310 (+ 626us) server_negotiation.cc:407] Connection header received
0621 02:44:11.518690 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:11.518701 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:11.518970 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:11.519147 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:11.520229 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.522092 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.525339 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.527175 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.527294 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:11.527589 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:11.528409 (+ 820us) server_negotiation.cc:299] Negotiation successful
0621 02:44:11.528627 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":576,"threads_started":1}
I20260621 02:44:11.530238 10924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:11.516750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:36201 (local address 127.0.0.1:55076)
0621 02:44:11.518176 (+ 1426us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:11.518220 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:11.518456 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:11.519538 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:11.519548 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:11.519593 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:11.520003 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:11.520018 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.522301 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.522309 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:11.525136 (+ 2827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:11.525154 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.527793 (+ 2639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.527798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:11.528105 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:11.528187 (+ 82us) client_negotiation.cc:769] Sending connection context
0621 02:44:11.529696 (+ 1509us) client_negotiation.cc:240] Negotiation successful
0621 02:44:11.529888 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1172,"thread_start_us":12879,"threads_started":1}
I20260621 02:44:11.536299 10578 tablet_service.cc:1467] Processing CreateTablet for tablet 4a86e00763ce4e79a206917337c1a167 (DEFAULT_TABLE table=test-table [id=93bc6ed61a2a4971b95d701c629191fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:11.539690 10578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a86e00763ce4e79a206917337c1a167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:11.541968 10924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:11.517432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:41903 (local address 127.0.0.1:33420)
0621 02:44:11.530623 (+ 13191us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:11.530639 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:11.530721 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:11.534924 (+ 4203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:11.534928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:11.534953 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:11.535133 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:11.535144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.537100 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.537104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:11.538990 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:11.539003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.541118 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.541125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:11.541358 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:11.541395 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 02:44:11.541465 (+ 70us) client_negotiation.cc:240] Negotiation successful
0621 02:44:11.541654 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13067,"thread_start_us":18060,"threads_started":1}
I20260621 02:44:11.543085 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:11.517777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33420 (local address 127.9.166.3:41903)
0621 02:44:11.534312 (+ 16535us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:11.534321 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:11.534351 (+ 30us) server_negotiation.cc:407] Connection header received
0621 02:44:11.534451 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:11.534459 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:11.534712 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:11.534936 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:11.535317 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.536913 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.539220 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.540905 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.542379 (+ 1474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:11.542562 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:11.542655 (+ 93us) server_negotiation.cc:299] Negotiation successful
0621 02:44:11.542824 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16356,"thread_start_us":17359,"threads_started":1}
I20260621 02:44:11.544569 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:11.517128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:42771 (local address 127.0.0.1:53970)
0621 02:44:11.529997 (+ 12869us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:11.530032 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:11.530190 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:11.536566 (+ 6376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:11.536571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:11.536596 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:11.536942 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:11.536957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.539079 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.539088 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:11.542099 (+ 3011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:11.542114 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.543937 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.543943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:11.544123 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:11.544154 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 02:44:11.544248 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 02:44:11.544352 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12700,"thread_start_us":925,"threads_started":1}
I20260621 02:44:11.545205 10925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:11.517397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53970 (local address 127.9.166.2:42771)
0621 02:44:11.536028 (+ 18631us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:11.536036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:11.536061 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:44:11.536155 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:11.536162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:11.536361 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:11.536537 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:11.537130 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.538895 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.542289 (+ 3394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.543739 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.543879 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:11.544064 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:11.544802 (+ 738us) server_negotiation.cc:299] Negotiation successful
0621 02:44:11.544976 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18476,"thread_start_us":18347,"threads_started":1}
I20260621 02:44:11.547073 10845 tablet_service.cc:1467] Processing CreateTablet for tablet 4a86e00763ce4e79a206917337c1a167 (DEFAULT_TABLE table=test-table [id=93bc6ed61a2a4971b95d701c629191fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:11.548550 10845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a86e00763ce4e79a206917337c1a167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:11.548337 10712 tablet_service.cc:1467] Processing CreateTablet for tablet 4a86e00763ce4e79a206917337c1a167 (DEFAULT_TABLE table=test-table [id=93bc6ed61a2a4971b95d701c629191fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:11.549443 10712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a86e00763ce4e79a206917337c1a167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:11.555702 10928 tablet_bootstrap.cc:492] T 4a86e00763ce4e79a206917337c1a167 P df26c5f8d4db4278b0d74abfdd69a570: Bootstrap starting.
I20260621 02:44:11.559377 10928 tablet_bootstrap.cc:654] T 4a86e00763ce4e79a206917337c1a167 P df26c5f8d4db4278b0d74abfdd69a570: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:11.560789 10928 log.cc:824] T 4a86e00763ce4e79a206917337c1a167 P df26c5f8d4db4278b0d74abfdd69a570: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:11.561329 10929 tablet_bootstrap.cc:492] T 4a86e00763ce4e79a206917337c1a167 P 818b3a86f93348e99800cea085333056: Bootstrap starting.
I20260621 02:44:11.564172 10930 tablet_bootstrap.cc:492] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c: Bootstrap starting.
I20260621 02:44:11.564253 10929 tablet_bootstrap.cc:654] T 4a86e00763ce4e79a206917337c1a167 P 818b3a86f93348e99800cea085333056: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:11.565411 10929 log.cc:824] T 4a86e00763ce4e79a206917337c1a167 P 818b3a86f93348e99800cea085333056: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:11.566907 10928 tablet_bootstrap.cc:492] T 4a86e00763ce4e79a206917337c1a167 P df26c5f8d4db4278b0d74abfdd69a570: No bootstrap required, opened a new log
I20260621 02:44:11.567224 10928 ts_tablet_manager.cc:1397] T 4a86e00763ce4e79a206917337c1a167 P df26c5f8d4db4278b0d74abfdd69a570: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260621 02:44:11.568084 10930 tablet_bootstrap.cc:654] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:11.569229 10929 tablet_bootstrap.cc:492] T 4a86e00763ce4e79a206917337c1a167 P 818b3a86f93348e99800cea085333056: No bootstrap required, opened a new log
I20260621 02:44:11.569329 10930 log.cc:824] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:11.569511 10929 ts_tablet_manager.cc:1397] T 4a86e00763ce4e79a206917337c1a167 P 818b3a86f93348e99800cea085333056: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 02:44:11.572346 10930 tablet_bootstrap.cc:492] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c: No bootstrap required, opened a new log
I20260621 02:44:11.572603 10930 ts_tablet_manager.cc:1397] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260621 02:44:11.576854 10929 raft_consensus.cc:348] T 4a86e00763ce4e79a206917337c1a167 P 818b3a86f93348e99800cea085333056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df26c5f8d4db4278b0d74abfdd69a570" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36201 } } peers { permanent_uuid: "818b3a86f93348e99800cea085333056" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42771 } } peers { permanent_uuid: "6439f40d119742bd996e861f4b4d062c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41903 } }
I20260621 02:44:11.577195 10929 raft_consensus.cc:374] T 4a86e00763ce4e79a206917337c1a167 P 818b3a86f93348e99800cea085333056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:11.577299 10929 raft_consensus.cc:729] T 4a86e00763ce4e79a206917337c1a167 P 818b3a86f93348e99800cea085333056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818b3a86f93348e99800cea085333056, State: Initialized, Role: FOLLOWER
I20260621 02:44:11.577517 10928 raft_consensus.cc:348] T 4a86e00763ce4e79a206917337c1a167 P df26c5f8d4db4278b0d74abfdd69a570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df26c5f8d4db4278b0d74abfdd69a570" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36201 } } peers { permanent_uuid: "818b3a86f93348e99800cea085333056" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42771 } } peers { permanent_uuid: "6439f40d119742bd996e861f4b4d062c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41903 } }
I20260621 02:44:11.577921 10928 raft_consensus.cc:374] T 4a86e00763ce4e79a206917337c1a167 P df26c5f8d4db4278b0d74abfdd69a570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:11.578056 10928 raft_consensus.cc:729] T 4a86e00763ce4e79a206917337c1a167 P df26c5f8d4db4278b0d74abfdd69a570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df26c5f8d4db4278b0d74abfdd69a570, State: Initialized, Role: FOLLOWER
I20260621 02:44:11.577950 10929 consensus_queue.cc:260] T 4a86e00763ce4e79a206917337c1a167 P 818b3a86f93348e99800cea085333056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df26c5f8d4db4278b0d74abfdd69a570" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36201 } } peers { permanent_uuid: "818b3a86f93348e99800cea085333056" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42771 } } peers { permanent_uuid: "6439f40d119742bd996e861f4b4d062c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41903 } }
I20260621 02:44:11.578696 10928 consensus_queue.cc:260] T 4a86e00763ce4e79a206917337c1a167 P df26c5f8d4db4278b0d74abfdd69a570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df26c5f8d4db4278b0d74abfdd69a570" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36201 } } peers { permanent_uuid: "818b3a86f93348e99800cea085333056" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42771 } } peers { permanent_uuid: "6439f40d119742bd996e861f4b4d062c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41903 } }
I20260621 02:44:11.584085 10777 heartbeater.cc:503] Master 127.9.166.62:34135 was elected leader, sending a full tablet report...
I20260621 02:44:11.584403 10930 raft_consensus.cc:348] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df26c5f8d4db4278b0d74abfdd69a570" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36201 } } peers { permanent_uuid: "818b3a86f93348e99800cea085333056" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42771 } } peers { permanent_uuid: "6439f40d119742bd996e861f4b4d062c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41903 } }
I20260621 02:44:11.584833 10930 raft_consensus.cc:374] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:11.584932 10930 raft_consensus.cc:729] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6439f40d119742bd996e861f4b4d062c, State: Initialized, Role: FOLLOWER
I20260621 02:44:11.585585 10929 ts_tablet_manager.cc:1428] T 4a86e00763ce4e79a206917337c1a167 P 818b3a86f93348e99800cea085333056: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260621 02:44:11.585489 10930 consensus_queue.cc:260] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df26c5f8d4db4278b0d74abfdd69a570" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36201 } } peers { permanent_uuid: "818b3a86f93348e99800cea085333056" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42771 } } peers { permanent_uuid: "6439f40d119742bd996e861f4b4d062c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41903 } }
I20260621 02:44:11.586743 10928 ts_tablet_manager.cc:1428] T 4a86e00763ce4e79a206917337c1a167 P df26c5f8d4db4278b0d74abfdd69a570: Time spent starting tablet: real 0.019s user 0.012s sys 0.005s
I20260621 02:44:11.587965 10936 raft_consensus.cc:482] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:44:11.587996 10910 heartbeater.cc:503] Master 127.9.166.62:34135 was elected leader, sending a full tablet report...
I20260621 02:44:11.588356 10936 raft_consensus.cc:504] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df26c5f8d4db4278b0d74abfdd69a570" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36201 } } peers { permanent_uuid: "818b3a86f93348e99800cea085333056" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42771 } } peers { permanent_uuid: "6439f40d119742bd996e861f4b4d062c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41903 } }
I20260621 02:44:11.588784 10930 ts_tablet_manager.cc:1428] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c: Time spent starting tablet: real 0.016s user 0.002s sys 0.014s
I20260621 02:44:11.592563 10936 leader_election.cc:288] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df26c5f8d4db4278b0d74abfdd69a570 (127.9.166.1:36201), 818b3a86f93348e99800cea085333056 (127.9.166.2:42771)
I20260621 02:44:11.597923 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:11.591176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:37625 (local address 127.9.166.1:36201)
0621 02:44:11.591727 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:11.591737 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:11.591761 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:44:11.591838 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:11.591845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:11.591937 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:11.592046 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:11.592851 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.593916 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.595974 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.597181 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.597306 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:11.597396 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:11.597672 (+ 276us) server_negotiation.cc:299] Negotiation successful
0621 02:44:11.597757 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369}
I20260621 02:44:11.598035 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:11.590677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:36201 (local address 127.9.166.3:37625)
0621 02:44:11.591170 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:11.591189 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:11.591317 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:11.592427 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:11.592434 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:11.592468 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:11.592715 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:11.592728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.594043 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.594049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:11.595847 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:11.595859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.597328 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.597333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:11.597516 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:11.597550 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 02:44:11.597648 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 02:44:11.597777 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":2}
I20260621 02:44:11.598994 10598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a86e00763ce4e79a206917337c1a167" candidate_uuid: "6439f40d119742bd996e861f4b4d062c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df26c5f8d4db4278b0d74abfdd69a570" is_pre_election: true
I20260621 02:44:11.599458 10925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:11.591466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:43281 (local address 127.9.166.2:42771)
0621 02:44:11.592346 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:11.592354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:11.592662 (+ 308us) server_negotiation.cc:407] Connection header received
0621 02:44:11.592948 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:11.592954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:11.593023 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:11.593124 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:11.593795 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.594824 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.597323 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.598604 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.598703 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:11.598764 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:11.599137 (+ 373us) server_negotiation.cc:299] Negotiation successful
0621 02:44:11.599219 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722}
I20260621 02:44:11.599677 10938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:11.590768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:42771 (local address 127.9.166.3:43281)
0621 02:44:11.592563 (+ 1795us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:11.592576 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:11.592764 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:11.593152 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:11.593157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:11.593182 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:11.593507 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:11.593518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.594983 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.594988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:11.597139 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:11.597154 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.598794 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.598800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:11.598973 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:11.599011 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 02:44:11.599119 (+ 108us) client_negotiation.cc:240] Negotiation successful
0621 02:44:11.599251 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1659,"thread_start_us":1506,"threads_started":1}
I20260621 02:44:11.599589 10598 raft_consensus.cc:2454] T 4a86e00763ce4e79a206917337c1a167 P df26c5f8d4db4278b0d74abfdd69a570 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6439f40d119742bd996e861f4b4d062c in term 0.
I20260621 02:44:11.600242 10800 leader_election.cc:302] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6439f40d119742bd996e861f4b4d062c, df26c5f8d4db4278b0d74abfdd69a570; no voters:
I20260621 02:44:11.600600 10936 raft_consensus.cc:2781] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:44:11.600754 10936 raft_consensus.cc:482] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:44:11.600652 10732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a86e00763ce4e79a206917337c1a167" candidate_uuid: "6439f40d119742bd996e861f4b4d062c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818b3a86f93348e99800cea085333056" is_pre_election: true
I20260621 02:44:11.600867 10936 raft_consensus.cc:3037] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:11.601178 10732 raft_consensus.cc:2454] T 4a86e00763ce4e79a206917337c1a167 P 818b3a86f93348e99800cea085333056 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6439f40d119742bd996e861f4b4d062c in term 0.
I20260621 02:44:11.603358 10936 raft_consensus.cc:504] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df26c5f8d4db4278b0d74abfdd69a570" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36201 } } peers { permanent_uuid: "818b3a86f93348e99800cea085333056" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42771 } } peers { permanent_uuid: "6439f40d119742bd996e861f4b4d062c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41903 } }
I20260621 02:44:11.604050 10936 leader_election.cc:288] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c [CANDIDATE]: Term 1 election: Requested vote from peers df26c5f8d4db4278b0d74abfdd69a570 (127.9.166.1:36201), 818b3a86f93348e99800cea085333056 (127.9.166.2:42771)
I20260621 02:44:11.604470 10598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a86e00763ce4e79a206917337c1a167" candidate_uuid: "6439f40d119742bd996e861f4b4d062c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df26c5f8d4db4278b0d74abfdd69a570"
I20260621 02:44:11.604470 10732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a86e00763ce4e79a206917337c1a167" candidate_uuid: "6439f40d119742bd996e861f4b4d062c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818b3a86f93348e99800cea085333056"
I20260621 02:44:11.604661 10732 raft_consensus.cc:3037] T 4a86e00763ce4e79a206917337c1a167 P 818b3a86f93348e99800cea085333056 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:11.604665 10598 raft_consensus.cc:3037] T 4a86e00763ce4e79a206917337c1a167 P df26c5f8d4db4278b0d74abfdd69a570 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:11.607827 10732 raft_consensus.cc:2454] T 4a86e00763ce4e79a206917337c1a167 P 818b3a86f93348e99800cea085333056 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6439f40d119742bd996e861f4b4d062c in term 1.
I20260621 02:44:11.607827 10598 raft_consensus.cc:2454] T 4a86e00763ce4e79a206917337c1a167 P df26c5f8d4db4278b0d74abfdd69a570 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6439f40d119742bd996e861f4b4d062c in term 1.
I20260621 02:44:11.608212 10800 leader_election.cc:302] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6439f40d119742bd996e861f4b4d062c, df26c5f8d4db4278b0d74abfdd69a570; no voters:
I20260621 02:44:11.608486 10936 raft_consensus.cc:2781] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:11.608640 10936 raft_consensus.cc:686] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c [term 1 LEADER]: Becoming Leader. State: Replica: 6439f40d119742bd996e861f4b4d062c, State: Running, Role: LEADER
I20260621 02:44:11.609079 10936 consensus_queue.cc:237] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df26c5f8d4db4278b0d74abfdd69a570" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36201 } } peers { permanent_uuid: "818b3a86f93348e99800cea085333056" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42771 } } peers { permanent_uuid: "6439f40d119742bd996e861f4b4d062c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41903 } }
I20260621 02:44:11.613029 10453 catalog_manager.cc:5495] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6439f40d119742bd996e861f4b4d062c (127.9.166.3). New cstate: current_term: 1 leader_uuid: "6439f40d119742bd996e861f4b4d062c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df26c5f8d4db4278b0d74abfdd69a570" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818b3a86f93348e99800cea085333056" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6439f40d119742bd996e861f4b4d062c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41903 } health_report { overall_health: HEALTHY } } }
Jun 21 02:44:11 dist-test-slave-hvgn krb5kdc[10409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:44:11.644042 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:11.629334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.9.166.1:36201)
0621 02:44:11.629502 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:11.629506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:11.630004 (+ 498us) server_negotiation.cc:407] Connection header received
0621 02:44:11.630126 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:11.630130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:11.630231 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:11.630311 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:11.632039 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.632821 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.634084 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.634504 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.634601 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:11.638443 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:11.638454 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:11.638469 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:11.638498 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:11.641417 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:11.642111 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:11.642118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:11.642120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:11.642247 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:11.642571 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:11.642575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:11.642577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:11.643011 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:11.643250 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:11.643503 (+ 253us) server_negotiation.cc:299] Negotiation successful
0621 02:44:11.643681 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Jun 21 02:44:11 dist-test-slave-hvgn krb5kdc[10409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009851, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 02:44:11 dist-test-slave-hvgn krb5kdc[10409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009851, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:44:11.687103 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:11.673654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.9.166.1:36201)
0621 02:44:11.673890 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:11.673896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:11.674114 (+ 218us) server_negotiation.cc:407] Connection header received
0621 02:44:11.674332 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:11.674336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:11.674398 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:11.674518 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:11.675976 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.676671 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.677838 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.678239 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.678327 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:11.681895 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:11.681900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:11.681903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:11.681932 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:11.684604 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:11.685257 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:11.685263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:11.685266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:11.685403 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:11.685773 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:11.685778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:11.685781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:11.686034 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:11.686298 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:11.686559 (+ 261us) server_negotiation.cc:299] Negotiation successful
0621 02:44:11.686770 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260621 02:44:11.694042 10911 tablet.cc:2369] T 4a86e00763ce4e79a206917337c1a167 P 6439f40d119742bd996e861f4b4d062c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 21 02:44:11 dist-test-slave-hvgn krb5kdc[10409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260621 02:44:11.714846 10778 tablet.cc:2369] T 4a86e00763ce4e79a206917337c1a167 P 818b3a86f93348e99800cea085333056: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 21 02:44:11 dist-test-slave-hvgn krb5kdc[10409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:44:11.729132 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:11.716244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55092 (local address 127.9.166.1:36201)
0621 02:44:11.716496 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:11.716503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:11.716747 (+ 244us) server_negotiation.cc:407] Connection header received
0621 02:44:11.716897 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:11.716900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:11.716964 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:11.717048 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:11.718287 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.718945 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.720168 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.720581 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.720692 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:11.724297 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:11.724302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:11.724306 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:11.724337 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:11.726815 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:11.727493 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:11.727500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:11.727508 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:11.727680 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:11.728005 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:11.728009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:11.728011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:11.728273 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:11.728451 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:11.728702 (+ 251us) server_negotiation.cc:299] Negotiation successful
0621 02:44:11.728871 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260621 02:44:11.730760 10558 tablet_service.cc:2908] Rejecting scan request for tablet 4a86e00763ce4e79a206917337c1a167: Uninitialized: clean time has not yet been initialized
I20260621 02:44:11.733994 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 10515
W20260621 02:44:11.737771 10644 tablet.cc:2369] T 4a86e00763ce4e79a206917337c1a167 P df26c5f8d4db4278b0d74abfdd69a570: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:44:11.746304 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:11.734822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55102 (local address 127.9.166.1:36201)
0621 02:44:11.734983 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:11.734987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:11.735363 (+ 376us) server_negotiation.cc:407] Connection header received
0621 02:44:11.735541 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:11.735545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:11.735607 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:11.735721 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:11.736774 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.737433 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.738940 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.739331 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.739504 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:11.741689 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:11.741693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:11.741696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:11.741721 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:11.744253 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:11.744810 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:11.744813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:11.744815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:11.744918 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:11.745201 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:11.745204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:11.745205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:11.745436 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:11.745626 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:11.745822 (+ 196us) server_negotiation.cc:299] Negotiation successful
0621 02:44:11.745967 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 02:44:11.746922 10638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:11.836412 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 10515
I20260621 02:44:11.853435 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 10649
Jun 21 02:44:11 dist-test-slave-hvgn krb5kdc[10409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.2@KRBTEST.COM
I20260621 02:44:11.868278 10925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:11.854250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53978 (local address 127.9.166.2:42771)
0621 02:44:11.854430 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:11.854436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:11.854483 (+ 47us) server_negotiation.cc:407] Connection header received
0621 02:44:11.854706 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:11.854710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:11.854796 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:11.854919 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:11.856346 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.857085 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.858461 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.858859 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.858988 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:11.862912 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:11.862926 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:11.862941 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:11.862972 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:11.865620 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:11.866268 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:11.866276 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:11.866278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:11.866401 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:11.866774 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:11.866781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:11.866784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:11.867151 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:11.867599 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:11.867746 (+ 147us) server_negotiation.cc:299] Negotiation successful
0621 02:44:11.867978 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:44:11.868903 10772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:11.955181 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 10649
I20260621 02:44:11.970820 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 10782
Jun 21 02:44:11 dist-test-slave-hvgn krb5kdc[10409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.3@KRBTEST.COM
I20260621 02:44:11.985601 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:11.971663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33434 (local address 127.9.166.3:41903)
0621 02:44:11.971846 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:11.971851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:11.971868 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:44:11.972008 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:11.972013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:11.972074 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:11.972156 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:11.973331 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.974029 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.975205 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:11.975655 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:11.975751 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:11.979642 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:11.979656 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:11.979672 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:11.979719 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:11.982273 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:11.982877 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:11.982884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:11.982886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:11.983013 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:11.983969 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:11.983974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:11.983978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:11.984391 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:11.984656 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:11.985007 (+ 351us) server_negotiation.cc:299] Negotiation successful
0621 02:44:11.985244 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:44:11.986218 10905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:12.074301 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 10782
I20260621 02:44:12.090322 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 10422
I20260621 02:44:12.091576 10484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:12.197875 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 10422
2026-06-21T02:44:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3987 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:44:12 dist-test-slave-hvgn krb5kdc[10957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:44:12 dist-test-slave-hvgn krb5kdc[10957](info): set up 2 sockets
Jun 21 02:44:12 dist-test-slave-hvgn krb5kdc[10957](info): commencing operation
krb5kdc: starting...
W20260621 02:44:13.291285 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:44:13 dist-test-slave-hvgn krb5kdc[10957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:44:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:44:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:44:13.462529 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:34155
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:34155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:13.772233 10970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:13.772533 10970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:13.772697 10970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:13.779865 10970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:44:13.779958 10970 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 02:44:13.780045 10970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:13.780160 10970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:13.780246 10970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:44:13.780329 10970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:44:13.789904 10970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:34155
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:34155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:13.791090 10970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:13.792395 10970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:13.803820 10978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:13.803987 10975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:13.803946 10976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:13.805095 10970 server_base.cc:1034] running on GCE node
I20260621 02:44:13.805819 10970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:13.807173 10970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:13.808405 10970 hybrid_clock.cc:648] HybridClock initialized: now 1782009853808328 us; error 56 us; skew 500 ppm
I20260621 02:44:13.808732 10970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:13 dist-test-slave-hvgn krb5kdc[10957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009853, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:13.811685 10970 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:44:13.813066 10970 webserver.cc:460] Webserver started at http://127.9.166.62:33837/ using document root <none> and password file <none>
I20260621 02:44:13.813709 10970 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:13.813792 10970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:13.814044 10970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:13.817024 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "1b7a3845d3ed41028340970bdf39db98"
format_stamp: "Formatted at 2026-06-21 02:44:13 on dist-test-slave-hvgn"
server_key: "2e6cb6f712e706b1bf113b9a12c0aab7"
server_key_iv: "02fc4a78fadeeb434da4a4f447516bce"
server_key_version: "encryptionkey@0"
I20260621 02:44:13.817749 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "1b7a3845d3ed41028340970bdf39db98"
format_stamp: "Formatted at 2026-06-21 02:44:13 on dist-test-slave-hvgn"
server_key: "2e6cb6f712e706b1bf113b9a12c0aab7"
server_key_iv: "02fc4a78fadeeb434da4a4f447516bce"
server_key_version: "encryptionkey@0"
I20260621 02:44:13.822589 10970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260621 02:44:13.825907 10985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:13.827265 10970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:44:13.827461 10970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "1b7a3845d3ed41028340970bdf39db98"
format_stamp: "Formatted at 2026-06-21 02:44:13 on dist-test-slave-hvgn"
server_key: "2e6cb6f712e706b1bf113b9a12c0aab7"
server_key_iv: "02fc4a78fadeeb434da4a4f447516bce"
server_key_version: "encryptionkey@0"
I20260621 02:44:13.827682 10970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:13.840723 10970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:13.844501 10970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:13.844770 10970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:13.866199 10970 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:34155
I20260621 02:44:13.866205 11046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:34155 every 8 connection(s)
I20260621 02:44:13.867626 10970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:44:13.871376 11047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:13.878017 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 10970
I20260621 02:44:13.878250 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:44:13.878892 9880 external_mini_cluster.cc:1428] Setting key 04469cdd38cd2c9b953b11b038ea809d
I20260621 02:44:13.881974 11047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98: Bootstrap starting.
I20260621 02:44:13.885917 11047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:13.887144 11047 log.cc:824] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:13.889851 11047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98: No bootstrap required, opened a new log
Jun 21 02:44:13 dist-test-slave-hvgn krb5kdc[10957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:13.899016 11047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7a3845d3ed41028340970bdf39db98" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34155 } }
I20260621 02:44:13.899415 11047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:13.899559 11047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b7a3845d3ed41028340970bdf39db98, State: Initialized, Role: FOLLOWER
I20260621 02:44:13.900177 11047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7a3845d3ed41028340970bdf39db98" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34155 } }
I20260621 02:44:13.900466 11047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:44:13.900611 11047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:44:13.900781 11047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:13.903419 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:13.880875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41306 (local address 127.9.166.62:34155)
0621 02:44:13.881682 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:13.881697 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:13.881749 (+ 52us) server_negotiation.cc:407] Connection header received
0621 02:44:13.884198 (+ 2449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:13.884226 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:13.884692 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:13.885101 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:13.886393 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:13.888239 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:13.889476 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:13.890482 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:13.890657 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:13.894398 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:13.894408 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:13.894426 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:13.894469 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:13.898728 (+ 4259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:13.899468 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:13.899475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:13.899481 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:13.899773 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:13.900217 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:13.900222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:13.900225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:13.900952 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:13.901231 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:13.901611 (+ 380us) server_negotiation.cc:299] Negotiation successful
0621 02:44:13.902054 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":333,"threads_started":1}
I20260621 02:44:13.904042 11047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7a3845d3ed41028340970bdf39db98" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34155 } }
I20260621 02:44:13.904619 11047 leader_election.cc:302] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b7a3845d3ed41028340970bdf39db98; no voters:
I20260621 02:44:13.905333 11047 leader_election.cc:288] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:44:13.905491 11052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:13.906291 11052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98 [term 1 LEADER]: Becoming Leader. State: Replica: 1b7a3845d3ed41028340970bdf39db98, State: Running, Role: LEADER
I20260621 02:44:13.906911 11052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7a3845d3ed41028340970bdf39db98" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34155 } }
I20260621 02:44:13.909336 11052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b7a3845d3ed41028340970bdf39db98. Latest consensus state: current_term: 1 leader_uuid: "1b7a3845d3ed41028340970bdf39db98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7a3845d3ed41028340970bdf39db98" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34155 } } }
I20260621 02:44:13.909662 11052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:13.909781 11054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b7a3845d3ed41028340970bdf39db98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7a3845d3ed41028340970bdf39db98" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34155 } } }
I20260621 02:44:13.910185 11054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:13.910815 11047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:44:13.913493 11056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:44:13.919178 11056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:44:13.927817 11056 catalog_manager.cc:1347] Generated new cluster ID: 371579f32b0e47b9a345eb57c8bef2b3
I20260621 02:44:13.927956 11056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:44:13.943085 11056 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:44:13.944916 11056 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:44:13.951954 11056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98: Generated new TSK 0
I20260621 02:44:13.952924 11056 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
I20260621 02:44:14.002938 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34155
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:14.318277 11073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:14.318652 11073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:14.318830 11073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:14.326009 11073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:14.326148 11073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:14.326292 11073 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:44:14.326426 11073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:44:14.336339 11073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.166.62:34155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:14.337581 11073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:14.339030 11073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:14.352102 11078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:14.352082 11081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:14.352048 11079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:14.352854 11073 server_base.cc:1034] running on GCE node
I20260621 02:44:14.353372 11073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:14.354199 11073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:14.355471 11073 hybrid_clock.cc:648] HybridClock initialized: now 1782009854355342 us; error 49 us; skew 500 ppm
I20260621 02:44:14.355767 11073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:14 dist-test-slave-hvgn krb5kdc[10957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009854, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:14.358906 11073 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:44:14.360435 11073 webserver.cc:460] Webserver started at http://127.9.166.1:35823/ using document root <none> and password file <none>
I20260621 02:44:14.361068 11073 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:14.361151 11073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:14.361404 11073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:14.364203 11073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "de7c7c91e26c44208e5999fa2738dfda"
format_stamp: "Formatted at 2026-06-21 02:44:14 on dist-test-slave-hvgn"
server_key: "0ba6828eaa47a4428b1b167774706299"
server_key_iv: "33f9433f30edfdd3bb01e5ea2d8902b4"
server_key_version: "encryptionkey@0"
I20260621 02:44:14.364945 11073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "de7c7c91e26c44208e5999fa2738dfda"
format_stamp: "Formatted at 2026-06-21 02:44:14 on dist-test-slave-hvgn"
server_key: "0ba6828eaa47a4428b1b167774706299"
server_key_iv: "33f9433f30edfdd3bb01e5ea2d8902b4"
server_key_version: "encryptionkey@0"
I20260621 02:44:14.369828 11073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:44:14.373174 11088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:14.374430 11073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:44:14.374679 11073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "de7c7c91e26c44208e5999fa2738dfda"
format_stamp: "Formatted at 2026-06-21 02:44:14 on dist-test-slave-hvgn"
server_key: "0ba6828eaa47a4428b1b167774706299"
server_key_iv: "33f9433f30edfdd3bb01e5ea2d8902b4"
server_key_version: "encryptionkey@0"
I20260621 02:44:14.374887 11073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:14.387856 11073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:14.391969 11073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:14.392239 11073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:14.394919 11073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:14.395026 11073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:14.395243 11073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:14.395358 11073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 21 02:44:14 dist-test-slave-hvgn krb5kdc[10957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009854, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:14.418617 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:14.395542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:38149 (local address 127.9.166.62:34155)
0621 02:44:14.395750 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:14.395756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:14.397083 (+ 1327us) server_negotiation.cc:407] Connection header received
0621 02:44:14.399079 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:14.399085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:14.399188 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:14.399353 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:14.402277 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:14.403199 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:14.406500 (+ 3301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:14.407042 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:14.407191 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:14.412016 (+ 4825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:14.412023 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:14.412027 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:14.412082 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:14.415088 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:14.416047 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:14.416053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:14.416056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:14.416204 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:14.416734 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:14.416739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:14.416743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:14.417009 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:14.417171 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:14.418044 (+ 873us) server_negotiation.cc:299] Negotiation successful
0621 02:44:14.418255 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:44:14.420089 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:14.396052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34155 (local address 127.9.166.1:38149)
0621 02:44:14.396897 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:14.396953 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:14.398801 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:14.399956 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:14.399969 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:14.400950 (+ 981us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:14.402052 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:14.402085 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:14.403417 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:14.403426 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:14.406312 (+ 2886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:14.406332 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:14.407351 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:14.407362 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:14.407717 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:14.408678 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:14.408738 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:14.411683 (+ 2945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:14.415255 (+ 3572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:14.415264 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:14.415288 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:14.415868 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:14.416361 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:14.416371 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:14.416374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:14.416609 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:14.417154 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:14.417161 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:14.417618 (+ 457us) client_negotiation.cc:769] Sending connection context
0621 02:44:14.417966 (+ 348us) client_negotiation.cc:240] Negotiation successful
0621 02:44:14.418526 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":375,"threads_started":1}
I20260621 02:44:14.441064 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:14.425723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:51071 (local address 127.9.166.62:34155)
0621 02:44:14.425937 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:14.425946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:14.426522 (+ 576us) server_negotiation.cc:407] Connection header received
0621 02:44:14.426704 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:14.426711 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:14.426799 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:14.426923 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:14.428357 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:14.429596 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:14.431626 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:14.432366 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:14.432506 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:14.434973 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:14.434977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:14.434980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:14.435008 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:14.438148 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:14.438838 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:14.438845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:14.438850 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:14.438979 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:14.439377 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:14.439418 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:14.439421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:14.439900 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:14.440053 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:14.440444 (+ 391us) server_negotiation.cc:299] Negotiation successful
0621 02:44:14.440676 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:44:14.441152 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:14.425699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34155 (local address 127.9.166.1:51071)
0621 02:44:14.426397 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:14.426412 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:14.426574 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:14.426949 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:14.426954 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:14.427671 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:14.428168 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:14.428188 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:14.429729 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:14.429737 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:14.431480 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:14.431500 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:14.432575 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:14.432583 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:14.432819 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:14.433493 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:14.433538 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:14.434798 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:14.438273 (+ 3475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:14.438282 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:14.438287 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:14.438697 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:14.439087 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:14.439095 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:14.439099 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:14.439262 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:14.440007 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:14.440013 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:14.440207 (+ 194us) client_negotiation.cc:769] Sending connection context
0621 02:44:14.440349 (+ 142us) client_negotiation.cc:240] Negotiation successful
0621 02:44:14.440676 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":384,"threads_started":1}
I20260621 02:44:14.450655 11073 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:37681
I20260621 02:44:14.450695 11207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:37681 every 8 connection(s)
I20260621 02:44:14.452059 11073 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:44:14.454126 11208 heartbeater.cc:347] Connected to a master server at 127.9.166.62:34155
I20260621 02:44:14.454358 11208 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:14.454981 11208 heartbeater.cc:511] Master 127.9.166.62:34155 requested a full tablet report, sending...
I20260621 02:44:14.456931 11002 ts_manager.cc:194] Registered new tserver with Master: de7c7c91e26c44208e5999fa2738dfda (127.9.166.1:37681)
I20260621 02:44:14.458314 11002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:38149
I20260621 02:44:14.461005 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 11073
I20260621 02:44:14.461220 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:44:14.461829 9880 external_mini_cluster.cc:1428] Setting key 218ca8a4806d8e68a1313c5d5e5a48b3
WARNING: no policy specified for kudu/127.9.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
I20260621 02:44:14.498524 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34155
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:14.811985 11212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:14.812299 11212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:14.812475 11212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:14.819878 11212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:14.820006 11212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:14.820163 11212 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:44:14.820293 11212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:44:14.830030 11212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.166.62:34155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:14.831396 11212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:14.832737 11212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:14.845265 11218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:14.845340 11217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:14.845336 11220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:14.846491 11212 server_base.cc:1034] running on GCE node
I20260621 02:44:14.847081 11212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:14.847970 11212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:14.849193 11212 hybrid_clock.cc:648] HybridClock initialized: now 1782009854849102 us; error 63 us; skew 500 ppm
I20260621 02:44:14.849483 11212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:14 dist-test-slave-hvgn krb5kdc[10957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009854, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:14.857019 11212 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:44:14.858587 11212 webserver.cc:460] Webserver started at http://127.9.166.2:41837/ using document root <none> and password file <none>
I20260621 02:44:14.859264 11212 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:14.859375 11212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:14.859746 11212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:14.863240 11212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "a5e1b6dbbd564152b350307b57f5fc06"
format_stamp: "Formatted at 2026-06-21 02:44:14 on dist-test-slave-hvgn"
server_key: "a93aa16e1d709c7063725431dea23b2b"
server_key_iv: "bd1ae5283fda5e9495a06c1572e77fdc"
server_key_version: "encryptionkey@0"
I20260621 02:44:14.864107 11212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "a5e1b6dbbd564152b350307b57f5fc06"
format_stamp: "Formatted at 2026-06-21 02:44:14 on dist-test-slave-hvgn"
server_key: "a93aa16e1d709c7063725431dea23b2b"
server_key_iv: "bd1ae5283fda5e9495a06c1572e77fdc"
server_key_version: "encryptionkey@0"
I20260621 02:44:14.869160 11212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:44:14.872624 11227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:14.873920 11212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:44:14.874083 11212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "a5e1b6dbbd564152b350307b57f5fc06"
format_stamp: "Formatted at 2026-06-21 02:44:14 on dist-test-slave-hvgn"
server_key: "a93aa16e1d709c7063725431dea23b2b"
server_key_iv: "bd1ae5283fda5e9495a06c1572e77fdc"
server_key_version: "encryptionkey@0"
I20260621 02:44:14.874284 11212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:14.900741 11212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:14.905021 11212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:14.905311 11212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:14.908063 11212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:14.908176 11212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:14.908360 11212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:14.908437 11212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:44:14 dist-test-slave-hvgn krb5kdc[10957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009854, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:14.934122 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:14.908646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:51115 (local address 127.9.166.62:34155)
0621 02:44:14.908824 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:14.908829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:14.910420 (+ 1591us) server_negotiation.cc:407] Connection header received
0621 02:44:14.913605 (+ 3185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:14.913615 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:14.913715 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:14.913836 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:14.916435 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:14.917402 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:14.920738 (+ 3336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:14.921188 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:14.921297 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:14.926468 (+ 5171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:14.926472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:14.926476 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:14.926509 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:14.930300 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:14.931365 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:14.931373 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:14.931377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:14.931534 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:14.932129 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:14.932132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:14.932134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:14.932492 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:14.932604 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:14.933678 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0621 02:44:14.933880 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260621 02:44:14.936261 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:14.909277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34155 (local address 127.9.166.2:51115)
0621 02:44:14.910205 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:14.910280 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:14.913216 (+ 2936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:14.914143 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:14.914158 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:14.915218 (+ 1060us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:14.916230 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:14.916271 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:14.917610 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:14.917622 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:14.920557 (+ 2935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:14.920578 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:14.921443 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:14.921454 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:14.921815 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:14.922666 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:14.922706 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:14.926156 (+ 3450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:14.930486 (+ 4330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:14.930495 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:14.930515 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:14.931247 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:14.931653 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:14.931662 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:14.931666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:14.931966 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:14.932633 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:14.932646 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:14.933209 (+ 563us) client_negotiation.cc:769] Sending connection context
0621 02:44:14.933589 (+ 380us) client_negotiation.cc:240] Negotiation successful
0621 02:44:14.934288 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":415,"threads_started":1}
I20260621 02:44:14.958374 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:14.943983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:54349 (local address 127.9.166.62:34155)
0621 02:44:14.944182 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:14.944189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:14.945628 (+ 1439us) server_negotiation.cc:407] Connection header received
0621 02:44:14.945766 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:14.945775 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:14.945880 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:14.946004 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:14.947102 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:14.948043 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:14.949972 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:14.950491 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:14.950620 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:14.952681 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:14.952687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:14.952692 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:14.952731 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:14.955427 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:14.956321 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:14.956329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:14.956335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:14.956500 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:14.956984 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:14.956987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:14.956989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:14.957302 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:14.957414 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:14.957940 (+ 526us) server_negotiation.cc:299] Negotiation successful
0621 02:44:14.958103 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 02:44:14.958913 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:14.944332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34155 (local address 127.9.166.2:54349)
0621 02:44:14.945503 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:14.945527 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:14.945645 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:14.946044 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:14.946047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:14.946631 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:14.946959 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:14.946968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:14.948218 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:14.948221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:14.949824 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:14.949839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:14.950690 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:14.950696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:14.950847 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:14.951504 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:14.951541 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:14.952510 (+ 969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:14.955566 (+ 3056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:14.955571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:14.955575 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:14.956191 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:14.956635 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:14.956643 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:14.956649 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:14.956845 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:14.957420 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:14.957428 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:14.957676 (+ 248us) client_negotiation.cc:769] Sending connection context
0621 02:44:14.957867 (+ 191us) client_negotiation.cc:240] Negotiation successful
0621 02:44:14.958282 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"thread_start_us":856,"threads_started":1}
I20260621 02:44:14.973021 11212 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:42511
I20260621 02:44:14.973052 11346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:42511 every 8 connection(s)
I20260621 02:44:14.974184 11212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:44:14.976104 11347 heartbeater.cc:347] Connected to a master server at 127.9.166.62:34155
I20260621 02:44:14.976339 11347 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:14.976971 11347 heartbeater.cc:511] Master 127.9.166.62:34155 requested a full tablet report, sending...
I20260621 02:44:14.978375 11002 ts_manager.cc:194] Registered new tserver with Master: a5e1b6dbbd564152b350307b57f5fc06 (127.9.166.2:42511)
I20260621 02:44:14.979281 11002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:51115
I20260621 02:44:14.984278 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 11212
I20260621 02:44:14.984439 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:44:14.984815 9880 external_mini_cluster.cc:1428] Setting key 83108b44375ab65a49587e1bf4881101
WARNING: no policy specified for kudu/127.9.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
I20260621 02:44:15.022356 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34155
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:15.334302 11351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:15.334561 11351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:15.334712 11351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:15.342069 11351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:15.342186 11351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:15.342338 11351 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:44:15.342463 11351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:44:15.351958 11351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.166.62:34155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:15.353134 11351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:15.354485 11351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:15.366987 11356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:15.366987 11359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:15.366976 11357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:15.368129 11351 server_base.cc:1034] running on GCE node
I20260621 02:44:15.368676 11351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:15.369544 11351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:15.370777 11351 hybrid_clock.cc:648] HybridClock initialized: now 1782009855370721 us; error 31 us; skew 500 ppm
I20260621 02:44:15.371097 11351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:15 dist-test-slave-hvgn krb5kdc[10957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009855, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:15.374145 11351 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:44:15.375648 11351 webserver.cc:460] Webserver started at http://127.9.166.3:46425/ using document root <none> and password file <none>
I20260621 02:44:15.376367 11351 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:15.376492 11351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:15.376803 11351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:15.379942 11351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "cca8c39ad4db4bc59c244e008a3dacea"
format_stamp: "Formatted at 2026-06-21 02:44:15 on dist-test-slave-hvgn"
server_key: "ec4859f39ea29cef0b585c4d5af8ee4f"
server_key_iv: "b5a13e737600a7f0bdab53aa58787a95"
server_key_version: "encryptionkey@0"
I20260621 02:44:15.380870 11351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "cca8c39ad4db4bc59c244e008a3dacea"
format_stamp: "Formatted at 2026-06-21 02:44:15 on dist-test-slave-hvgn"
server_key: "ec4859f39ea29cef0b585c4d5af8ee4f"
server_key_iv: "b5a13e737600a7f0bdab53aa58787a95"
server_key_version: "encryptionkey@0"
I20260621 02:44:15.385946 11351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:44:15.389346 11366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:15.390590 11351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:44:15.390756 11351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "cca8c39ad4db4bc59c244e008a3dacea"
format_stamp: "Formatted at 2026-06-21 02:44:15 on dist-test-slave-hvgn"
server_key: "ec4859f39ea29cef0b585c4d5af8ee4f"
server_key_iv: "b5a13e737600a7f0bdab53aa58787a95"
server_key_version: "encryptionkey@0"
I20260621 02:44:15.390961 11351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:15.409981 11351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:15.414075 11351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:15.414459 11351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:15.417605 11351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:15.417712 11351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:15.417878 11351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:15.417972 11351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:44:15 dist-test-slave-hvgn krb5kdc[10957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009855, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:15.442037 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:15.418092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:50249 (local address 127.9.166.62:34155)
0621 02:44:15.418333 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:15.418343 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:15.419846 (+ 1503us) server_negotiation.cc:407] Connection header received
0621 02:44:15.422866 (+ 3020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:15.422874 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:15.422994 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:15.423094 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:15.425991 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.427058 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.430306 (+ 3248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.430933 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.431076 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:15.435021 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:15.435026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:15.435030 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:15.435063 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:15.438409 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:15.439478 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:15.439486 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:15.439489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:15.439672 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:15.440204 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:15.440210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:15.440215 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:15.440631 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:15.440791 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:15.441448 (+ 657us) server_negotiation.cc:299] Negotiation successful
0621 02:44:15.441675 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:44:15.443668 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:15.418729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34155 (local address 127.9.166.3:50249)
0621 02:44:15.419659 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:15.419715 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:15.422496 (+ 2781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:15.423709 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:15.423724 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:15.424737 (+ 1013us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:15.425761 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:15.425799 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.427246 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.427255 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:15.430109 (+ 2854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:15.430128 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.431228 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.431238 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:15.431537 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:15.432333 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:15.432355 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:15.434766 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:15.438565 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:15.438577 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:15.438594 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:15.439329 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:15.439826 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:15.439831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:15.439835 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:15.440076 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:15.440764 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:15.440772 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:15.441107 (+ 335us) client_negotiation.cc:769] Sending connection context
0621 02:44:15.441358 (+ 251us) client_negotiation.cc:240] Negotiation successful
0621 02:44:15.441984 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":376,"threads_started":1}
I20260621 02:44:15.460773 11208 heartbeater.cc:503] Master 127.9.166.62:34155 was elected leader, sending a full tablet report...
I20260621 02:44:15.474637 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:15.460222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:37057 (local address 127.9.166.62:34155)
0621 02:44:15.460416 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:15.460424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:15.460711 (+ 287us) server_negotiation.cc:407] Connection header received
0621 02:44:15.460905 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:15.460911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:15.460996 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:15.461098 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:15.462139 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.463593 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.465189 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.466131 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.466287 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:15.468691 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:15.468697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:15.468704 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:15.468739 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:15.472143 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:15.472755 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:15.472762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:15.472766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:15.472909 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:15.473262 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:15.473268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:15.473271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:15.473509 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:15.473648 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:15.474041 (+ 393us) server_negotiation.cc:299] Negotiation successful
0621 02:44:15.474258 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 02:44:15.474726 11461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:15.459771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34155 (local address 127.9.166.3:37057)
0621 02:44:15.460597 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:15.460615 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:15.460773 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:15.461120 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:15.461124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:15.461614 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:15.461986 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:15.461998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.463746 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.463751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:15.465053 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:15.465064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.466315 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.466319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:15.466435 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:15.467125 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:15.467156 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:15.468431 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:15.472267 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:15.472272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:15.472276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:15.472640 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:15.473018 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:15.473024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:15.473027 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:15.473157 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:15.473629 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:15.473634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:15.473810 (+ 176us) client_negotiation.cc:769] Sending connection context
0621 02:44:15.473950 (+ 140us) client_negotiation.cc:240] Negotiation successful
0621 02:44:15.474251 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":653,"threads_started":1}
I20260621 02:44:15.477825 11351 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:40503
I20260621 02:44:15.477869 11485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:40503 every 8 connection(s)
I20260621 02:44:15.479171 11351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:44:15.481155 11486 heartbeater.cc:347] Connected to a master server at 127.9.166.62:34155
I20260621 02:44:15.481331 11486 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:15.481873 11486 heartbeater.cc:511] Master 127.9.166.62:34155 requested a full tablet report, sending...
I20260621 02:44:15.483080 11002 ts_manager.cc:194] Registered new tserver with Master: cca8c39ad4db4bc59c244e008a3dacea (127.9.166.3:40503)
I20260621 02:44:15.483865 11002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:50249
I20260621 02:44:15.488004 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 11351
I20260621 02:44:15.488157 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:44:15.488536 9880 external_mini_cluster.cc:1428] Setting key c66273d9b488b6c52172766770d2c465
I20260621 02:44:15.491153 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 02:44:15 dist-test-slave-hvgn krb5kdc[10957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009855, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 02:44:15 dist-test-slave-hvgn krb5kdc[10957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009855, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:15.534798 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:15.520993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41316 (local address 127.9.166.62:34155)
0621 02:44:15.521217 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:15.521225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:15.521453 (+ 228us) server_negotiation.cc:407] Connection header received
0621 02:44:15.521709 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:15.521713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:15.521772 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:15.521873 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:15.523288 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.524136 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.525395 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.525804 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.525934 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:15.529708 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:15.529712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:15.529716 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:15.529750 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:15.532390 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:15.532978 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:15.532982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:15.532985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:15.533105 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:15.533468 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:15.533472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:15.533479 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:15.533733 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:15.533980 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:15.534247 (+ 267us) server_negotiation.cc:299] Negotiation successful
0621 02:44:15.534471 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:44:15.538921 11002 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:44:15.546099 11002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:44:15.578634 11501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:15.559036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53350 (local address 127.9.166.1:37681)
0621 02:44:15.571357 (+ 12321us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:15.571367 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:15.571412 (+ 45us) server_negotiation.cc:407] Connection header received
0621 02:44:15.571522 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:15.571529 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:15.571735 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:15.571929 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:15.572560 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.573964 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.576425 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.577382 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.577582 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:15.577802 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:15.578164 (+ 362us) server_negotiation.cc:299] Negotiation successful
0621 02:44:15.578327 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12174,"thread_start_us":12049,"threads_started":1}
I20260621 02:44:15.578711 11500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:15.558733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:37681 (local address 127.0.0.1:53350)
0621 02:44:15.569858 (+ 11125us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:15.569919 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:15.570091 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:15.571934 (+ 1843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:15.571941 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:15.571990 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:15.572383 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:15.572392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.574093 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.574101 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:15.576274 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:15.576285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.577724 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.577730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:15.577917 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:15.577976 (+ 59us) client_negotiation.cc:769] Sending connection context
0621 02:44:15.578103 (+ 127us) client_negotiation.cc:240] Negotiation successful
0621 02:44:15.578283 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10942,"mutex_wait_us":353,"spinlock_wait_cycles":3328,"thread_start_us":10817,"threads_started":1}
I20260621 02:44:15.580366 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:15.558392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:40503 (local address 127.0.0.1:55386)
0621 02:44:15.569858 (+ 11466us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:15.571306 (+ 1448us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:15.571454 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:15.572254 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:15.572265 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:15.572289 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:15.572626 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:15.572633 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.574843 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.574868 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:15.577476 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:15.577490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.579560 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.579566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:15.579716 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:15.579756 (+ 40us) client_negotiation.cc:769] Sending connection context
0621 02:44:15.579862 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 02:44:15.579988 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11195,"mutex_wait_us":13,"thread_start_us":12397,"threads_started":1}
I20260621 02:44:15.580674 11499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:15.559462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:42511 (local address 127.0.0.1:48044)
0621 02:44:15.571062 (+ 11600us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:15.571080 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:15.571194 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:15.571938 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:15.571943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:15.571991 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:15.573138 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:15.573152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.574961 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.574966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:15.577597 (+ 2631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:15.577610 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.579846 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.579852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:15.580028 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:15.580063 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 02:44:15.580151 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 02:44:15.580239 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11472,"mutex_wait_us":548,"spinlock_wait_cycles":3712,"thread_start_us":9967,"threads_started":1}
I20260621 02:44:15.581498 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:15.560048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48044 (local address 127.9.166.2:42511)
0621 02:44:15.570254 (+ 10206us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:15.570264 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:15.571301 (+ 1037us) server_negotiation.cc:407] Connection header received
0621 02:44:15.571443 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:15.571451 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:15.571717 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:15.572089 (+ 372us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:15.573355 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.574789 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.577808 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.579349 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.579558 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:15.579719 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:15.580484 (+ 765us) server_negotiation.cc:299] Negotiation successful
0621 02:44:15.580667 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10034,"thread_start_us":10918,"threads_started":1}
I20260621 02:44:15.582660 11138 tablet_service.cc:1467] Processing CreateTablet for tablet 6c938ae4d1d14b9590b392689349567f (DEFAULT_TABLE table=test-table [id=88fc6c92a3f74f36bd61b7d65d891108]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:15.583897 11138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c938ae4d1d14b9590b392689349567f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:15.584195 11277 tablet_service.cc:1467] Processing CreateTablet for tablet 6c938ae4d1d14b9590b392689349567f (DEFAULT_TABLE table=test-table [id=88fc6c92a3f74f36bd61b7d65d891108]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:15.585232 11277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c938ae4d1d14b9590b392689349567f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:15.588236 11502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:15.559264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55386 (local address 127.9.166.3:40503)
0621 02:44:15.570847 (+ 11583us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:15.570856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:15.571433 (+ 577us) server_negotiation.cc:407] Connection header received
0621 02:44:15.571600 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:15.571607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:15.571777 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:15.571963 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:15.572819 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.574649 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.577632 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.579331 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.579509 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:15.579775 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:15.579903 (+ 128us) server_negotiation.cc:299] Negotiation successful
0621 02:44:15.587907 (+ 8004us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11395,"thread_start_us":11190,"threads_started":1}
I20260621 02:44:15.591641 11416 tablet_service.cc:1467] Processing CreateTablet for tablet 6c938ae4d1d14b9590b392689349567f (DEFAULT_TABLE table=test-table [id=88fc6c92a3f74f36bd61b7d65d891108]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:15.592955 11416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c938ae4d1d14b9590b392689349567f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:15.594838 11505 tablet_bootstrap.cc:492] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda: Bootstrap starting.
I20260621 02:44:15.598199 11505 tablet_bootstrap.cc:654] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:15.599166 11505 log.cc:824] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:15.600469 11506 tablet_bootstrap.cc:492] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06: Bootstrap starting.
I20260621 02:44:15.602278 11505 tablet_bootstrap.cc:492] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda: No bootstrap required, opened a new log
I20260621 02:44:15.602540 11505 ts_tablet_manager.cc:1397] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260621 02:44:15.603849 11506 tablet_bootstrap.cc:654] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:15.605052 11506 log.cc:824] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:15.607890 11506 tablet_bootstrap.cc:492] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06: No bootstrap required, opened a new log
I20260621 02:44:15.607905 11509 tablet_bootstrap.cc:492] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea: Bootstrap starting.
I20260621 02:44:15.608137 11506 ts_tablet_manager.cc:1397] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260621 02:44:15.610330 11509 tablet_bootstrap.cc:654] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:15.611313 11509 log.cc:824] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:15.612980 11505 raft_consensus.cc:348] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } } peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } }
I20260621 02:44:15.613399 11505 raft_consensus.cc:374] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:15.613492 11505 raft_consensus.cc:729] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7c7c91e26c44208e5999fa2738dfda, State: Initialized, Role: FOLLOWER
I20260621 02:44:15.613823 11509 tablet_bootstrap.cc:492] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea: No bootstrap required, opened a new log
I20260621 02:44:15.614074 11509 ts_tablet_manager.cc:1397] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260621 02:44:15.614066 11505 consensus_queue.cc:260] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } } peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } }
I20260621 02:44:15.615744 11505 ts_tablet_manager.cc:1428] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260621 02:44:15.618242 11506 raft_consensus.cc:348] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } } peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } }
I20260621 02:44:15.618664 11506 raft_consensus.cc:374] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:15.618772 11506 raft_consensus.cc:729] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e1b6dbbd564152b350307b57f5fc06, State: Initialized, Role: FOLLOWER
I20260621 02:44:15.619306 11506 consensus_queue.cc:260] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } } peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } }
I20260621 02:44:15.621512 11347 heartbeater.cc:503] Master 127.9.166.62:34155 was elected leader, sending a full tablet report...
I20260621 02:44:15.622097 11506 ts_tablet_manager.cc:1428] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260621 02:44:15.623916 11509 raft_consensus.cc:348] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } } peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } }
I20260621 02:44:15.624221 11509 raft_consensus.cc:374] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:15.624341 11509 raft_consensus.cc:729] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca8c39ad4db4bc59c244e008a3dacea, State: Initialized, Role: FOLLOWER
I20260621 02:44:15.624905 11509 consensus_queue.cc:260] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } } peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } }
I20260621 02:44:15.625854 11486 heartbeater.cc:503] Master 127.9.166.62:34155 was elected leader, sending a full tablet report...
I20260621 02:44:15.626353 11509 ts_tablet_manager.cc:1428] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea: Time spent starting tablet: real 0.012s user 0.004s sys 0.006s
W20260621 02:44:15.704555 11209 tablet.cc:2369] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:44:15.726193 11348 tablet.cc:2369] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:44:15.730806 11487 tablet.cc:2369] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:44:15.872232 11512 raft_consensus.cc:482] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:44:15.872614 11512 raft_consensus.cc:504] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } } peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } }
I20260621 02:44:15.874590 11512 leader_election.cc:288] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca8c39ad4db4bc59c244e008a3dacea (127.9.166.3:40503), de7c7c91e26c44208e5999fa2738dfda (127.9.166.1:37681)
I20260621 02:44:15.882846 11511 raft_consensus.cc:482] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:44:15.883203 11511 raft_consensus.cc:504] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } } peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } }
I20260621 02:44:15.883620 11501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:15.875822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:41887 (local address 127.9.166.1:37681)
0621 02:44:15.876000 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:15.876007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:15.877361 (+ 1354us) server_negotiation.cc:407] Connection header received
0621 02:44:15.877621 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:15.877628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:15.877728 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:15.877839 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:15.878430 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.879358 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.881236 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.882425 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.882574 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:15.882684 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:15.883011 (+ 327us) server_negotiation.cc:299] Negotiation successful
0621 02:44:15.883149 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260621 02:44:15.883808 11514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:15.875001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:37681 (local address 127.9.166.2:41887)
0621 02:44:15.877270 (+ 2269us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:15.877286 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:15.877396 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:15.877853 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:15.877859 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:15.877886 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:15.878264 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:15.878274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.879537 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.879542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:15.881086 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:15.881098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.882650 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.882656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:15.882823 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:15.882856 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 02:44:15.883245 (+ 389us) client_negotiation.cc:240] Negotiation successful
0621 02:44:15.883424 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2135,"mutex_wait_us":1,"thread_start_us":1168,"threads_started":1}
I20260621 02:44:15.883896 11502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:15.875187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:50947 (local address 127.9.166.3:40503)
0621 02:44:15.875373 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:15.875379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:15.876749 (+ 1370us) server_negotiation.cc:407] Connection header received
0621 02:44:15.876946 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:15.876954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:15.877050 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:15.877149 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:15.877933 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.879017 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.881522 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.882714 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.882840 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:15.882952 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:15.883522 (+ 570us) server_negotiation.cc:299] Negotiation successful
0621 02:44:15.883648 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:44:15.884563 11515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:15.874948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:40503 (local address 127.9.166.2:50947)
0621 02:44:15.876614 (+ 1666us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:15.876633 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:15.876784 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:15.877236 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:15.877240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:15.877273 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:15.877761 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:15.877777 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.879137 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.879142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:15.881361 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:15.881375 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.883102 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.883110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:15.883297 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:15.883344 (+ 47us) client_negotiation.cc:769] Sending connection context
0621 02:44:15.884066 (+ 722us) client_negotiation.cc:240] Negotiation successful
0621 02:44:15.884195 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1519,"spinlock_wait_cycles":27392,"thread_start_us":2047,"threads_started":1}
I20260621 02:44:15.884835 11163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c938ae4d1d14b9590b392689349567f" candidate_uuid: "a5e1b6dbbd564152b350307b57f5fc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7c7c91e26c44208e5999fa2738dfda" is_pre_election: true
I20260621 02:44:15.885316 11163 raft_consensus.cc:2454] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e1b6dbbd564152b350307b57f5fc06 in term 0.
I20260621 02:44:15.885419 11511 leader_election.cc:288] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca8c39ad4db4bc59c244e008a3dacea (127.9.166.3:40503), a5e1b6dbbd564152b350307b57f5fc06 (127.9.166.2:42511)
I20260621 02:44:15.885603 11436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c938ae4d1d14b9590b392689349567f" candidate_uuid: "a5e1b6dbbd564152b350307b57f5fc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca8c39ad4db4bc59c244e008a3dacea" is_pre_election: true
I20260621 02:44:15.886191 11436 raft_consensus.cc:2454] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e1b6dbbd564152b350307b57f5fc06 in term 0.
I20260621 02:44:15.886739 11229 leader_election.cc:302] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5e1b6dbbd564152b350307b57f5fc06, de7c7c91e26c44208e5999fa2738dfda; no voters:
I20260621 02:44:15.887451 11512 raft_consensus.cc:2781] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:44:15.887605 11512 raft_consensus.cc:482] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:44:15.887713 11512 raft_consensus.cc:3037] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:15.890362 11512 raft_consensus.cc:504] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } } peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } }
I20260621 02:44:15.891072 11512 leader_election.cc:288] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [CANDIDATE]: Term 1 election: Requested vote from peers cca8c39ad4db4bc59c244e008a3dacea (127.9.166.3:40503), de7c7c91e26c44208e5999fa2738dfda (127.9.166.1:37681)
I20260621 02:44:15.891435 11436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c938ae4d1d14b9590b392689349567f" candidate_uuid: "a5e1b6dbbd564152b350307b57f5fc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca8c39ad4db4bc59c244e008a3dacea"
I20260621 02:44:15.891624 11436 raft_consensus.cc:3037] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:15.891675 11163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c938ae4d1d14b9590b392689349567f" candidate_uuid: "a5e1b6dbbd564152b350307b57f5fc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7c7c91e26c44208e5999fa2738dfda"
I20260621 02:44:15.891853 11163 raft_consensus.cc:3037] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:15.894783 11436 raft_consensus.cc:2454] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e1b6dbbd564152b350307b57f5fc06 in term 1.
I20260621 02:44:15.895123 11163 raft_consensus.cc:2454] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e1b6dbbd564152b350307b57f5fc06 in term 1.
I20260621 02:44:15.895368 11502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:15.886273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:53667 (local address 127.9.166.3:40503)
0621 02:44:15.887972 (+ 1699us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:15.887977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:15.887993 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:44:15.888044 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:15.888048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:15.888116 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:15.888198 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:15.888947 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.890062 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.892882 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.894115 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.894207 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:15.894293 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:15.894973 (+ 680us) server_negotiation.cc:299] Negotiation successful
0621 02:44:15.895114 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1586}
I20260621 02:44:15.895413 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:15.886312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:49133 (local address 127.9.166.2:42511)
0621 02:44:15.886786 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:15.886794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:15.887576 (+ 782us) server_negotiation.cc:407] Connection header received
0621 02:44:15.887820 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:15.887825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:15.887895 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:15.887954 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:15.888859 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.890026 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.892890 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.894078 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.894177 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:15.894270 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:15.894997 (+ 727us) server_negotiation.cc:299] Negotiation successful
0621 02:44:15.895138 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299}
I20260621 02:44:15.895437 11231 leader_election.cc:302] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5e1b6dbbd564152b350307b57f5fc06, cca8c39ad4db4bc59c244e008a3dacea; no voters:
I20260621 02:44:15.896068 11516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:15.885726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:40503 (local address 127.9.166.1:53667)
0621 02:44:15.886432 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:15.886451 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:15.886560 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:15.888250 (+ 1690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:15.888254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:15.888295 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:15.888752 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:15.888765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.890209 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.890216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:15.892704 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:15.892717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.894524 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.894531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:15.894754 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:15.894806 (+ 52us) client_negotiation.cc:769] Sending connection context
0621 02:44:15.895523 (+ 717us) client_negotiation.cc:240] Negotiation successful
0621 02:44:15.895654 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":75,"thread_start_us":1981,"threads_started":1}
I20260621 02:44:15.896133 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:15.885907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:42511 (local address 127.9.166.1:49133)
0621 02:44:15.887485 (+ 1578us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:15.887505 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:15.887665 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:15.888148 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:15.888155 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:15.888190 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:15.888657 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:15.888674 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.890235 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.890239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:15.892711 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:15.892725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.894488 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.894497 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:15.894771 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:15.894824 (+ 53us) client_negotiation.cc:769] Sending connection context
0621 02:44:15.895575 (+ 751us) client_negotiation.cc:240] Negotiation successful
0621 02:44:15.895706 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1427,"mutex_wait_us":54,"thread_start_us":1239,"threads_started":1}
I20260621 02:44:15.896186 11512 raft_consensus.cc:2781] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:15.896871 11436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c938ae4d1d14b9590b392689349567f" candidate_uuid: "de7c7c91e26c44208e5999fa2738dfda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca8c39ad4db4bc59c244e008a3dacea" is_pre_election: true
I20260621 02:44:15.897142 11436 raft_consensus.cc:2379] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de7c7c91e26c44208e5999fa2738dfda in current term 1: Already voted for candidate a5e1b6dbbd564152b350307b57f5fc06 in this term.
I20260621 02:44:15.897225 11302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c938ae4d1d14b9590b392689349567f" candidate_uuid: "de7c7c91e26c44208e5999fa2738dfda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e1b6dbbd564152b350307b57f5fc06" is_pre_election: true
I20260621 02:44:15.897581 11512 raft_consensus.cc:686] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [term 1 LEADER]: Becoming Leader. State: Replica: a5e1b6dbbd564152b350307b57f5fc06, State: Running, Role: LEADER
I20260621 02:44:15.898106 11512 consensus_queue.cc:237] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } } peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } }
I20260621 02:44:15.901705 11089 leader_election.cc:302] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de7c7c91e26c44208e5999fa2738dfda; no voters: a5e1b6dbbd564152b350307b57f5fc06, cca8c39ad4db4bc59c244e008a3dacea
I20260621 02:44:15.902076 11511 raft_consensus.cc:2726] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260621 02:44:15.902505 11002 catalog_manager.cc:5495] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5e1b6dbbd564152b350307b57f5fc06 (127.9.166.2). New cstate: current_term: 1 leader_uuid: "a5e1b6dbbd564152b350307b57f5fc06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } health_report { overall_health: HEALTHY } } }
I20260621 02:44:15.939417 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:15.933684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48046 (local address 127.9.166.2:42511)
0621 02:44:15.933893 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:15.933900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:15.934028 (+ 128us) server_negotiation.cc:407] Connection header received
0621 02:44:15.934353 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:15.934357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:15.934441 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:15.934585 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:44:15.934972 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.935815 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.937101 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:15.937483 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:15.937625 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:15.938149 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:44:15.938729 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:44:15.938884 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:15.939006 (+ 122us) server_negotiation.cc:299] Negotiation successful
0621 02:44:15.939091 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:44:15.947273 11163 raft_consensus.cc:1261] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda [term 1 FOLLOWER]: Refusing update from remote peer a5e1b6dbbd564152b350307b57f5fc06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:44:15.947252 11436 raft_consensus.cc:1261] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea [term 1 FOLLOWER]: Refusing update from remote peer a5e1b6dbbd564152b350307b57f5fc06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:44:15.948089 11518 consensus_queue.cc:1035] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:15.948459 11512 consensus_queue.cc:1035] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:15.959376 11521 mvcc.cc:205] Tried to move back new op lower bound from 7299112369950117888 to 7299112369761873920. Current Snapshot: MvccSnapshot[applied={T|T < 7299112369950117888}]
I20260621 02:44:15.967046 11522 mvcc.cc:205] Tried to move back new op lower bound from 7299112369950117888 to 7299112369761873920. Current Snapshot: MvccSnapshot[applied={T|T < 7299112369950117888}]
I20260621 02:44:15.978111 11001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:44:15.978410 11001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:44:15.982308 11001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98: Sending DeleteTablet for 3 replicas of tablet 6c938ae4d1d14b9590b392689349567f
I20260621 02:44:15.983299 11138 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c938ae4d1d14b9590b392689349567f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:15 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:53350
I20260621 02:44:15.983310 11416 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c938ae4d1d14b9590b392689349567f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:15 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:55386
I20260621 02:44:15.984174 11531 tablet_replica.cc:330] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda: stopping tablet replica
I20260621 02:44:15.984524 11277 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c938ae4d1d14b9590b392689349567f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:15 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:48044
I20260621 02:44:15.984649 11531 raft_consensus.cc:2229] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:44:15.985107 11531 raft_consensus.cc:2258] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:15.987069 11001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:44:15.987704 11001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:44:15.994922 11414 tablet_service.cc:1467] Processing CreateTablet for tablet 66ba967ba60049ada25cb4cff71b4b82 (DEFAULT_TABLE table=test-table [id=be6e02ff1c954c28874f6668be537c89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:15.995051 11276 tablet_service.cc:1467] Processing CreateTablet for tablet 66ba967ba60049ada25cb4cff71b4b82 (DEFAULT_TABLE table=test-table [id=be6e02ff1c954c28874f6668be537c89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:15.995445 11414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ba967ba60049ada25cb4cff71b4b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:15.995637 11276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ba967ba60049ada25cb4cff71b4b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:15.996438 11533 tablet_replica.cc:330] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06: stopping tablet replica
I20260621 02:44:15.996881 11533 raft_consensus.cc:2229] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:44:15.997504 11533 raft_consensus.cc:2258] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:15.999743 11138 tablet_service.cc:1467] Processing CreateTablet for tablet 66ba967ba60049ada25cb4cff71b4b82 (DEFAULT_TABLE table=test-table [id=be6e02ff1c954c28874f6668be537c89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:16.000164 11138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ba967ba60049ada25cb4cff71b4b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:16.001307 11533 ts_tablet_manager.cc:1905] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:16.001711 11509 tablet_bootstrap.cc:492] T 66ba967ba60049ada25cb4cff71b4b82 P cca8c39ad4db4bc59c244e008a3dacea: Bootstrap starting.
I20260621 02:44:16.003297 11509 tablet_bootstrap.cc:654] T 66ba967ba60049ada25cb4cff71b4b82 P cca8c39ad4db4bc59c244e008a3dacea: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:16.004380 11532 tablet_replica.cc:330] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea: stopping tablet replica
I20260621 02:44:16.004902 11532 raft_consensus.cc:2229] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:44:16.005436 11532 raft_consensus.cc:2258] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:16.005887 11505 tablet_bootstrap.cc:492] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda: Bootstrap starting.
I20260621 02:44:16.007423 11505 tablet_bootstrap.cc:654] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:16.007853 11532 ts_tablet_manager.cc:1905] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:16.010686 11506 tablet_bootstrap.cc:492] T 66ba967ba60049ada25cb4cff71b4b82 P a5e1b6dbbd564152b350307b57f5fc06: Bootstrap starting.
I20260621 02:44:16.012924 11506 tablet_bootstrap.cc:654] T 66ba967ba60049ada25cb4cff71b4b82 P a5e1b6dbbd564152b350307b57f5fc06: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:16.019138 11533 ts_tablet_manager.cc:1918] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:44:16.019313 11533 log.cc:1196] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/6c938ae4d1d14b9590b392689349567f
I20260621 02:44:16.021734 11531 ts_tablet_manager.cc:1905] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:16.023633 11505 tablet_bootstrap.cc:492] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda: No bootstrap required, opened a new log
I20260621 02:44:16.023797 11505 ts_tablet_manager.cc:1397] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.001s
I20260621 02:44:16.023969 11533 ts_tablet_manager.cc:1939] T 6c938ae4d1d14b9590b392689349567f P a5e1b6dbbd564152b350307b57f5fc06: Deleting consensus metadata
I20260621 02:44:16.024271 11532 ts_tablet_manager.cc:1918] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:44:16.024428 11532 log.cc:1196] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/6c938ae4d1d14b9590b392689349567f
I20260621 02:44:16.024935 11532 ts_tablet_manager.cc:1939] T 6c938ae4d1d14b9590b392689349567f P cca8c39ad4db4bc59c244e008a3dacea: Deleting consensus metadata
I20260621 02:44:16.024821 11505 raft_consensus.cc:348] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } }
I20260621 02:44:16.025089 11505 raft_consensus.cc:374] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:16.025204 11505 raft_consensus.cc:729] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7c7c91e26c44208e5999fa2738dfda, State: Initialized, Role: FOLLOWER
I20260621 02:44:16.026103 11509 tablet_bootstrap.cc:492] T 66ba967ba60049ada25cb4cff71b4b82 P cca8c39ad4db4bc59c244e008a3dacea: No bootstrap required, opened a new log
I20260621 02:44:16.026311 11509 ts_tablet_manager.cc:1397] T 66ba967ba60049ada25cb4cff71b4b82 P cca8c39ad4db4bc59c244e008a3dacea: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.002s
I20260621 02:44:16.026418 11505 consensus_queue.cc:260] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } }
I20260621 02:44:16.026635 10989 catalog_manager.cc:4841] TS cca8c39ad4db4bc59c244e008a3dacea (127.9.166.3:40503): tablet 6c938ae4d1d14b9590b392689349567f (table test-table [id=88fc6c92a3f74f36bd61b7d65d891108]) successfully deleted
I20260621 02:44:16.027192 11505 ts_tablet_manager.cc:1428] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260621 02:44:16.027176 11509 raft_consensus.cc:348] T 66ba967ba60049ada25cb4cff71b4b82 P cca8c39ad4db4bc59c244e008a3dacea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } }
I20260621 02:44:16.027675 11511 raft_consensus.cc:482] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:44:16.027814 11511 raft_consensus.cc:504] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } }
I20260621 02:44:16.028158 11509 raft_consensus.cc:374] T 66ba967ba60049ada25cb4cff71b4b82 P cca8c39ad4db4bc59c244e008a3dacea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:16.028303 11509 raft_consensus.cc:729] T 66ba967ba60049ada25cb4cff71b4b82 P cca8c39ad4db4bc59c244e008a3dacea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca8c39ad4db4bc59c244e008a3dacea, State: Initialized, Role: FOLLOWER
I20260621 02:44:16.028446 11511 leader_election.cc:288] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5e1b6dbbd564152b350307b57f5fc06 (127.9.166.2:42511), cca8c39ad4db4bc59c244e008a3dacea (127.9.166.3:40503)
I20260621 02:44:16.028838 11302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ba967ba60049ada25cb4cff71b4b82" candidate_uuid: "de7c7c91e26c44208e5999fa2738dfda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e1b6dbbd564152b350307b57f5fc06" is_pre_election: true
I20260621 02:44:16.028859 11436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ba967ba60049ada25cb4cff71b4b82" candidate_uuid: "de7c7c91e26c44208e5999fa2738dfda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca8c39ad4db4bc59c244e008a3dacea" is_pre_election: true
I20260621 02:44:16.029457 11509 consensus_queue.cc:260] T 66ba967ba60049ada25cb4cff71b4b82 P cca8c39ad4db4bc59c244e008a3dacea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } }
W20260621 02:44:16.029608 11089 leader_election.cc:341] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5e1b6dbbd564152b350307b57f5fc06 (127.9.166.2:42511): Illegal state: must be running to vote when last-logged opid is not known
I20260621 02:44:16.029791 11436 raft_consensus.cc:2454] T 66ba967ba60049ada25cb4cff71b4b82 P cca8c39ad4db4bc59c244e008a3dacea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de7c7c91e26c44208e5999fa2738dfda in term 0.
I20260621 02:44:16.030222 11092 leader_election.cc:302] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cca8c39ad4db4bc59c244e008a3dacea, de7c7c91e26c44208e5999fa2738dfda; no voters: a5e1b6dbbd564152b350307b57f5fc06
I20260621 02:44:16.030694 11511 raft_consensus.cc:2781] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:44:16.030809 11511 raft_consensus.cc:482] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:44:16.030910 11511 raft_consensus.cc:3037] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:16.031512 11509 ts_tablet_manager.cc:1428] T 66ba967ba60049ada25cb4cff71b4b82 P cca8c39ad4db4bc59c244e008a3dacea: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260621 02:44:16.031622 10986 catalog_manager.cc:4841] TS a5e1b6dbbd564152b350307b57f5fc06 (127.9.166.2:42511): tablet 6c938ae4d1d14b9590b392689349567f (table test-table [id=88fc6c92a3f74f36bd61b7d65d891108]) successfully deleted
I20260621 02:44:16.032899 11506 tablet_bootstrap.cc:492] T 66ba967ba60049ada25cb4cff71b4b82 P a5e1b6dbbd564152b350307b57f5fc06: No bootstrap required, opened a new log
I20260621 02:44:16.033042 11506 ts_tablet_manager.cc:1397] T 66ba967ba60049ada25cb4cff71b4b82 P a5e1b6dbbd564152b350307b57f5fc06: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.003s
I20260621 02:44:16.033766 11506 raft_consensus.cc:348] T 66ba967ba60049ada25cb4cff71b4b82 P a5e1b6dbbd564152b350307b57f5fc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } }
I20260621 02:44:16.033999 11506 raft_consensus.cc:374] T 66ba967ba60049ada25cb4cff71b4b82 P a5e1b6dbbd564152b350307b57f5fc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:16.033949 11511 raft_consensus.cc:504] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } }
I20260621 02:44:16.034087 11506 raft_consensus.cc:729] T 66ba967ba60049ada25cb4cff71b4b82 P a5e1b6dbbd564152b350307b57f5fc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e1b6dbbd564152b350307b57f5fc06, State: Initialized, Role: FOLLOWER
I20260621 02:44:16.034427 11506 consensus_queue.cc:260] T 66ba967ba60049ada25cb4cff71b4b82 P a5e1b6dbbd564152b350307b57f5fc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } }
I20260621 02:44:16.035030 11506 ts_tablet_manager.cc:1428] T 66ba967ba60049ada25cb4cff71b4b82 P a5e1b6dbbd564152b350307b57f5fc06: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260621 02:44:16.036232 11531 ts_tablet_manager.cc:1918] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:44:16.036388 11531 log.cc:1196] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/6c938ae4d1d14b9590b392689349567f
I20260621 02:44:16.036896 11531 ts_tablet_manager.cc:1939] T 6c938ae4d1d14b9590b392689349567f P de7c7c91e26c44208e5999fa2738dfda: Deleting consensus metadata
I20260621 02:44:16.038401 11302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ba967ba60049ada25cb4cff71b4b82" candidate_uuid: "de7c7c91e26c44208e5999fa2738dfda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e1b6dbbd564152b350307b57f5fc06"
I20260621 02:44:16.038609 11302 raft_consensus.cc:3037] T 66ba967ba60049ada25cb4cff71b4b82 P a5e1b6dbbd564152b350307b57f5fc06 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:16.038736 11436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ba967ba60049ada25cb4cff71b4b82" candidate_uuid: "de7c7c91e26c44208e5999fa2738dfda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca8c39ad4db4bc59c244e008a3dacea"
I20260621 02:44:16.038918 11436 raft_consensus.cc:3037] T 66ba967ba60049ada25cb4cff71b4b82 P cca8c39ad4db4bc59c244e008a3dacea [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:16.039016 10987 catalog_manager.cc:4841] TS de7c7c91e26c44208e5999fa2738dfda (127.9.166.1:37681): tablet 6c938ae4d1d14b9590b392689349567f (table test-table [id=88fc6c92a3f74f36bd61b7d65d891108]) successfully deleted
I20260621 02:44:16.039260 11511 leader_election.cc:288] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [CANDIDATE]: Term 1 election: Requested vote from peers a5e1b6dbbd564152b350307b57f5fc06 (127.9.166.2:42511), cca8c39ad4db4bc59c244e008a3dacea (127.9.166.3:40503)
I20260621 02:44:16.041709 11436 raft_consensus.cc:2454] T 66ba967ba60049ada25cb4cff71b4b82 P cca8c39ad4db4bc59c244e008a3dacea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de7c7c91e26c44208e5999fa2738dfda in term 1.
I20260621 02:44:16.041776 11302 raft_consensus.cc:2454] T 66ba967ba60049ada25cb4cff71b4b82 P a5e1b6dbbd564152b350307b57f5fc06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de7c7c91e26c44208e5999fa2738dfda in term 1.
I20260621 02:44:16.042105 11092 leader_election.cc:302] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca8c39ad4db4bc59c244e008a3dacea, de7c7c91e26c44208e5999fa2738dfda; no voters:
I20260621 02:44:16.042521 11511 raft_consensus.cc:2781] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:16.042956 11511 raft_consensus.cc:686] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [term 1 LEADER]: Becoming Leader. State: Replica: de7c7c91e26c44208e5999fa2738dfda, State: Running, Role: LEADER
I20260621 02:44:16.043462 11511 consensus_queue.cc:237] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } }
I20260621 02:44:16.046854 11000 catalog_manager.cc:5495] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda reported cstate change: term changed from 0 to 1, leader changed from <none> to de7c7c91e26c44208e5999fa2738dfda (127.9.166.1). New cstate: current_term: 1 leader_uuid: "de7c7c91e26c44208e5999fa2738dfda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } health_report { overall_health: UNKNOWN } } }
Jun 21 02:44:16 dist-test-slave-hvgn krb5kdc[10957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009853, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:16.098871 11545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:16.085612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.9.166.62:34155)
0621 02:44:16.086310 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:16.086315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:16.086337 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:44:16.086380 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:16.086383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:16.086434 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:16.086555 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:16.087560 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.088328 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.089484 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.089865 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.089999 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:16.093549 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:16.093554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:16.093557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:16.093583 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:16.096266 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:16.096901 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:16.096908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:16.096914 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:16.097026 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:16.097393 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:16.097398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:16.097401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:16.097642 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:16.097784 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:16.098168 (+ 384us) server_negotiation.cc:299] Negotiation successful
0621 02:44:16.098501 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":584,"thread_start_us":477,"threads_started":1}
I20260621 02:44:16.098897 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:16.085529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34155 (local address 127.0.0.1:41320)
0621 02:44:16.085997 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:16.086025 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:16.086185 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:16.086593 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:16.086599 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:16.087190 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:16.087362 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:16.087370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.088467 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.088473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:16.089354 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:16.089365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.090013 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.090019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:16.090253 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:16.091010 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:16.091031 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:16.093352 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:16.096388 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:16.096402 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:16.096410 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:16.096788 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:16.097133 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:16.097140 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:16.097143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:16.097282 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:16.097759 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:16.097767 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:16.097926 (+ 159us) client_negotiation.cc:769] Sending connection context
0621 02:44:16.098096 (+ 170us) client_negotiation.cc:240] Negotiation successful
0621 02:44:16.098369 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":51,"thread_start_us":248,"threads_started":1}
I20260621 02:44:16.104745 11000 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:41320:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 02:44:16.105871 11000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:44:16.113101 11138 tablet_service.cc:1467] Processing CreateTablet for tablet baa43ac14a11470aa58b9d62eea247ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3254807d6f224b3596a4f6afd1cf5c35]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:44:16.113204 11276 tablet_service.cc:1467] Processing CreateTablet for tablet baa43ac14a11470aa58b9d62eea247ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3254807d6f224b3596a4f6afd1cf5c35]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:44:16.113682 11138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baa43ac14a11470aa58b9d62eea247ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:16.113761 11276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baa43ac14a11470aa58b9d62eea247ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:16.114387 11416 tablet_service.cc:1467] Processing CreateTablet for tablet baa43ac14a11470aa58b9d62eea247ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3254807d6f224b3596a4f6afd1cf5c35]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:44:16.114929 11416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baa43ac14a11470aa58b9d62eea247ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:16.122282 11505 tablet_bootstrap.cc:492] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda: Bootstrap starting.
I20260621 02:44:16.124190 11505 tablet_bootstrap.cc:654] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:16.125180 11509 tablet_bootstrap.cc:492] T baa43ac14a11470aa58b9d62eea247ad P cca8c39ad4db4bc59c244e008a3dacea: Bootstrap starting.
I20260621 02:44:16.126914 11506 tablet_bootstrap.cc:492] T baa43ac14a11470aa58b9d62eea247ad P a5e1b6dbbd564152b350307b57f5fc06: Bootstrap starting.
I20260621 02:44:16.127460 11509 tablet_bootstrap.cc:654] T baa43ac14a11470aa58b9d62eea247ad P cca8c39ad4db4bc59c244e008a3dacea: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:16.128873 11506 tablet_bootstrap.cc:654] T baa43ac14a11470aa58b9d62eea247ad P a5e1b6dbbd564152b350307b57f5fc06: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:16.129693 11509 tablet_bootstrap.cc:492] T baa43ac14a11470aa58b9d62eea247ad P cca8c39ad4db4bc59c244e008a3dacea: No bootstrap required, opened a new log
I20260621 02:44:16.129844 11509 ts_tablet_manager.cc:1397] T baa43ac14a11470aa58b9d62eea247ad P cca8c39ad4db4bc59c244e008a3dacea: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260621 02:44:16.131624 11509 raft_consensus.cc:348] T baa43ac14a11470aa58b9d62eea247ad P cca8c39ad4db4bc59c244e008a3dacea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } }
I20260621 02:44:16.131879 11509 raft_consensus.cc:374] T baa43ac14a11470aa58b9d62eea247ad P cca8c39ad4db4bc59c244e008a3dacea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:16.131801 11506 tablet_bootstrap.cc:492] T baa43ac14a11470aa58b9d62eea247ad P a5e1b6dbbd564152b350307b57f5fc06: No bootstrap required, opened a new log
I20260621 02:44:16.131956 11509 raft_consensus.cc:729] T baa43ac14a11470aa58b9d62eea247ad P cca8c39ad4db4bc59c244e008a3dacea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca8c39ad4db4bc59c244e008a3dacea, State: Initialized, Role: FOLLOWER
I20260621 02:44:16.132302 11506 ts_tablet_manager.cc:1397] T baa43ac14a11470aa58b9d62eea247ad P a5e1b6dbbd564152b350307b57f5fc06: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260621 02:44:16.132218 11509 consensus_queue.cc:260] T baa43ac14a11470aa58b9d62eea247ad P cca8c39ad4db4bc59c244e008a3dacea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } }
I20260621 02:44:16.132659 11513 tablet_replica.cc:439] T baa43ac14a11470aa58b9d62eea247ad P cca8c39ad4db4bc59c244e008a3dacea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } } }
I20260621 02:44:16.132989 11513 tablet_replica.cc:442] T baa43ac14a11470aa58b9d62eea247ad P cca8c39ad4db4bc59c244e008a3dacea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:44:16.133960 11506 raft_consensus.cc:348] T baa43ac14a11470aa58b9d62eea247ad P a5e1b6dbbd564152b350307b57f5fc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } }
I20260621 02:44:16.134455 11506 raft_consensus.cc:374] T baa43ac14a11470aa58b9d62eea247ad P a5e1b6dbbd564152b350307b57f5fc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:16.134625 11506 raft_consensus.cc:729] T baa43ac14a11470aa58b9d62eea247ad P a5e1b6dbbd564152b350307b57f5fc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e1b6dbbd564152b350307b57f5fc06, State: Initialized, Role: FOLLOWER
I20260621 02:44:16.135107 11506 consensus_queue.cc:260] T baa43ac14a11470aa58b9d62eea247ad P a5e1b6dbbd564152b350307b57f5fc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } }
I20260621 02:44:16.135802 11526 tablet_replica.cc:439] T baa43ac14a11470aa58b9d62eea247ad P a5e1b6dbbd564152b350307b57f5fc06: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } } }
I20260621 02:44:16.136307 11526 tablet_replica.cc:442] T baa43ac14a11470aa58b9d62eea247ad P a5e1b6dbbd564152b350307b57f5fc06: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:44:16.136559 11509 ts_tablet_manager.cc:1428] T baa43ac14a11470aa58b9d62eea247ad P cca8c39ad4db4bc59c244e008a3dacea: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260621 02:44:16.137424 11506 ts_tablet_manager.cc:1428] T baa43ac14a11470aa58b9d62eea247ad P a5e1b6dbbd564152b350307b57f5fc06: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260621 02:44:16.138818 11505 tablet_bootstrap.cc:492] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda: No bootstrap required, opened a new log
I20260621 02:44:16.139014 11505 ts_tablet_manager.cc:1397] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.010s
I20260621 02:44:16.139797 11505 raft_consensus.cc:348] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } }
I20260621 02:44:16.140075 11505 raft_consensus.cc:374] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:16.140185 11505 raft_consensus.cc:729] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7c7c91e26c44208e5999fa2738dfda, State: Initialized, Role: FOLLOWER
I20260621 02:44:16.140434 11505 consensus_queue.cc:260] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } }
I20260621 02:44:16.141076 11505 ts_tablet_manager.cc:1428] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260621 02:44:16.141323 11511 tablet_replica.cc:439] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } } }
I20260621 02:44:16.141608 11511 tablet_replica.cc:442] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:44:16.328727 11511 raft_consensus.cc:482] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:44:16.329005 11511 raft_consensus.cc:504] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } }
I20260621 02:44:16.329806 11511 leader_election.cc:288] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5e1b6dbbd564152b350307b57f5fc06 (127.9.166.2:42511), cca8c39ad4db4bc59c244e008a3dacea (127.9.166.3:40503)
I20260621 02:44:16.330415 11302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa43ac14a11470aa58b9d62eea247ad" candidate_uuid: "de7c7c91e26c44208e5999fa2738dfda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e1b6dbbd564152b350307b57f5fc06" is_pre_election: true
I20260621 02:44:16.330446 11436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa43ac14a11470aa58b9d62eea247ad" candidate_uuid: "de7c7c91e26c44208e5999fa2738dfda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca8c39ad4db4bc59c244e008a3dacea" is_pre_election: true
I20260621 02:44:16.330701 11302 raft_consensus.cc:2454] T baa43ac14a11470aa58b9d62eea247ad P a5e1b6dbbd564152b350307b57f5fc06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de7c7c91e26c44208e5999fa2738dfda in term 0.
I20260621 02:44:16.330758 11436 raft_consensus.cc:2454] T baa43ac14a11470aa58b9d62eea247ad P cca8c39ad4db4bc59c244e008a3dacea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de7c7c91e26c44208e5999fa2738dfda in term 0.
I20260621 02:44:16.331171 11089 leader_election.cc:302] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5e1b6dbbd564152b350307b57f5fc06, de7c7c91e26c44208e5999fa2738dfda; no voters:
I20260621 02:44:16.331645 11511 raft_consensus.cc:2781] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:44:16.331789 11511 raft_consensus.cc:482] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:44:16.331904 11511 raft_consensus.cc:3037] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:16.333851 11511 raft_consensus.cc:504] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } }
I20260621 02:44:16.334419 11511 leader_election.cc:288] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda [CANDIDATE]: Term 1 election: Requested vote from peers a5e1b6dbbd564152b350307b57f5fc06 (127.9.166.2:42511), cca8c39ad4db4bc59c244e008a3dacea (127.9.166.3:40503)
I20260621 02:44:16.334894 11436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa43ac14a11470aa58b9d62eea247ad" candidate_uuid: "de7c7c91e26c44208e5999fa2738dfda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca8c39ad4db4bc59c244e008a3dacea"
I20260621 02:44:16.334928 11302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa43ac14a11470aa58b9d62eea247ad" candidate_uuid: "de7c7c91e26c44208e5999fa2738dfda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e1b6dbbd564152b350307b57f5fc06"
I20260621 02:44:16.335144 11436 raft_consensus.cc:3037] T baa43ac14a11470aa58b9d62eea247ad P cca8c39ad4db4bc59c244e008a3dacea [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:16.335150 11302 raft_consensus.cc:3037] T baa43ac14a11470aa58b9d62eea247ad P a5e1b6dbbd564152b350307b57f5fc06 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:16.338112 11436 raft_consensus.cc:2454] T baa43ac14a11470aa58b9d62eea247ad P cca8c39ad4db4bc59c244e008a3dacea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de7c7c91e26c44208e5999fa2738dfda in term 1.
I20260621 02:44:16.338290 11302 raft_consensus.cc:2454] T baa43ac14a11470aa58b9d62eea247ad P a5e1b6dbbd564152b350307b57f5fc06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de7c7c91e26c44208e5999fa2738dfda in term 1.
I20260621 02:44:16.338640 11092 leader_election.cc:302] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca8c39ad4db4bc59c244e008a3dacea, de7c7c91e26c44208e5999fa2738dfda; no voters:
I20260621 02:44:16.339018 11511 raft_consensus.cc:2781] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:16.339376 11511 raft_consensus.cc:686] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda [term 1 LEADER]: Becoming Leader. State: Replica: de7c7c91e26c44208e5999fa2738dfda, State: Running, Role: LEADER
I20260621 02:44:16.339634 11511 consensus_queue.cc:237] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } }
I20260621 02:44:16.341377 11537 tablet_replica.cc:439] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda: TxnStatusTablet state changed. Reason: New leader de7c7c91e26c44208e5999fa2738dfda. Latest consensus state: current_term: 1 leader_uuid: "de7c7c91e26c44208e5999fa2738dfda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } } }
I20260621 02:44:16.341820 11537 tablet_replica.cc:442] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:44:16.342520 11550 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 02:44:16.342733 11550 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 02:44:16.343297 11000 catalog_manager.cc:5495] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda reported cstate change: term changed from 0 to 1, leader changed from <none> to de7c7c91e26c44208e5999fa2738dfda (127.9.166.1). New cstate: current_term: 1 leader_uuid: "de7c7c91e26c44208e5999fa2738dfda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } health_report { overall_health: UNKNOWN } } }
I20260621 02:44:16.378755 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:16.370924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:37681 (local address 127.0.0.1:53358)
0621 02:44:16.371263 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:16.371283 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:16.371450 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:16.372021 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:16.372027 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:16.372040 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:44:16.372216 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:16.372238 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.373721 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.373728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:16.375357 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:16.375373 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.376464 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.376469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:16.376720 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:16.376792 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:44:16.378001 (+ 1209us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:44:16.378021 (+ 20us) client_negotiation.cc:769] Sending connection context
0621 02:44:16.378186 (+ 165us) client_negotiation.cc:240] Negotiation successful
0621 02:44:16.378349 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260621 02:44:16.378834 11501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:16.370919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53358 (local address 127.9.166.1:37681)
0621 02:44:16.371117 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:16.371123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:16.371364 (+ 241us) server_negotiation.cc:407] Connection header received
0621 02:44:16.371698 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:16.371705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:16.371825 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:16.371984 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:44:16.372370 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.373575 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.375570 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.376211 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.376339 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:16.377000 (+ 661us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:44:16.377752 (+ 752us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:44:16.377938 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:16.378249 (+ 311us) server_negotiation.cc:299] Negotiation successful
0621 02:44:16.378384 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 02:44:16.383080 11436 raft_consensus.cc:1261] T baa43ac14a11470aa58b9d62eea247ad P cca8c39ad4db4bc59c244e008a3dacea [term 1 FOLLOWER]: Refusing update from remote peer de7c7c91e26c44208e5999fa2738dfda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:44:16.383419 11302 raft_consensus.cc:1261] T baa43ac14a11470aa58b9d62eea247ad P a5e1b6dbbd564152b350307b57f5fc06 [term 1 FOLLOWER]: Refusing update from remote peer de7c7c91e26c44208e5999fa2738dfda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:44:16.383813 11511 consensus_queue.cc:1035] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:16.384179 11537 consensus_queue.cc:1035] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:16.396431 11513 tablet_replica.cc:439] T baa43ac14a11470aa58b9d62eea247ad P cca8c39ad4db4bc59c244e008a3dacea: TxnStatusTablet state changed. Reason: New leader de7c7c91e26c44208e5999fa2738dfda. Latest consensus state: current_term: 1 leader_uuid: "de7c7c91e26c44208e5999fa2738dfda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } } }
I20260621 02:44:16.396703 11513 tablet_replica.cc:442] T baa43ac14a11470aa58b9d62eea247ad P cca8c39ad4db4bc59c244e008a3dacea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:44:16.398396 11526 tablet_replica.cc:439] T baa43ac14a11470aa58b9d62eea247ad P a5e1b6dbbd564152b350307b57f5fc06: TxnStatusTablet state changed. Reason: New leader de7c7c91e26c44208e5999fa2738dfda. Latest consensus state: current_term: 1 leader_uuid: "de7c7c91e26c44208e5999fa2738dfda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } } }
I20260621 02:44:16.398701 11526 tablet_replica.cc:442] T baa43ac14a11470aa58b9d62eea247ad P a5e1b6dbbd564152b350307b57f5fc06: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:44:16.400019 11511 tablet_replica.cc:439] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "de7c7c91e26c44208e5999fa2738dfda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } } }
I20260621 02:44:16.400259 11511 tablet_replica.cc:442] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:44:16.401858 11523 mvcc.cc:205] Tried to move back new op lower bound from 7299112371737083904 to 7299112371568738304. Current Snapshot: MvccSnapshot[applied={T|T < 7299112371737083904 or (T in {7299112371737083904})}]
I20260621 02:44:16.410094 11513 tablet_replica.cc:439] T baa43ac14a11470aa58b9d62eea247ad P cca8c39ad4db4bc59c244e008a3dacea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "de7c7c91e26c44208e5999fa2738dfda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } } }
I20260621 02:44:16.410369 11513 tablet_replica.cc:442] T baa43ac14a11470aa58b9d62eea247ad P cca8c39ad4db4bc59c244e008a3dacea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:44:16.416289 11537 tablet_replica.cc:439] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "de7c7c91e26c44208e5999fa2738dfda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } } }
I20260621 02:44:16.416548 11537 tablet_replica.cc:442] T baa43ac14a11470aa58b9d62eea247ad P de7c7c91e26c44208e5999fa2738dfda: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:44:16.419059 11526 tablet_replica.cc:439] T baa43ac14a11470aa58b9d62eea247ad P a5e1b6dbbd564152b350307b57f5fc06: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "de7c7c91e26c44208e5999fa2738dfda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de7c7c91e26c44208e5999fa2738dfda" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 37681 } } peers { permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 } } peers { permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 } } }
I20260621 02:44:16.419313 11526 tablet_replica.cc:442] T baa43ac14a11470aa58b9d62eea247ad P a5e1b6dbbd564152b350307b57f5fc06: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:44:16.421593 11501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:16.409889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53372 (local address 127.9.166.1:37681)
0621 02:44:16.412880 (+ 2991us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:16.412886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:16.412904 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:44:16.412963 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:16.412968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:16.413075 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:16.413646 (+ 571us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:44:16.413803 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.415116 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.417248 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.417923 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.418047 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:16.420622 (+ 2575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:44:16.420756 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:44:16.421207 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:16.421255 (+ 48us) server_negotiation.cc:299] Negotiation successful
0621 02:44:16.421328 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2886}
I20260621 02:44:16.440120 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:16.432829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:37681 (local address 127.9.166.1:45149)
0621 02:44:16.433515 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:16.433534 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:16.433677 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:16.434022 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:16.434026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:16.434038 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:44:16.434396 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:16.434406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.435913 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.435919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:16.437713 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:16.437726 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.438618 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.438625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:16.438821 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:16.438897 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:44:16.439339 (+ 442us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:44:16.439350 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 02:44:16.439530 (+ 180us) client_negotiation.cc:240] Negotiation successful
0621 02:44:16.439672 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":26,"thread_start_us":335,"threads_started":1}
I20260621 02:44:16.440193 11501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:16.432792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:45149 (local address 127.9.166.1:37681)
0621 02:44:16.432983 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:16.432989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:16.433621 (+ 632us) server_negotiation.cc:407] Connection header received
0621 02:44:16.433799 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:16.433808 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:16.433891 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:16.433998 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:44:16.434562 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.435782 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.437860 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.438468 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.438583 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:16.439071 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:44:16.439230 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:44:16.439322 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:16.439562 (+ 240us) server_negotiation.cc:299] Negotiation successful
0621 02:44:16.439869 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":57}
I20260621 02:44:16.451320 11436 raft_consensus.cc:1261] T 66ba967ba60049ada25cb4cff71b4b82 P cca8c39ad4db4bc59c244e008a3dacea [term 1 FOLLOWER]: Refusing update from remote peer de7c7c91e26c44208e5999fa2738dfda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:44:16.451897 11511 consensus_queue.cc:1035] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca8c39ad4db4bc59c244e008a3dacea" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:16.452854 11302 raft_consensus.cc:1261] T 66ba967ba60049ada25cb4cff71b4b82 P a5e1b6dbbd564152b350307b57f5fc06 [term 1 FOLLOWER]: Refusing update from remote peer de7c7c91e26c44208e5999fa2738dfda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:44:16.453395 11511 consensus_queue.cc:1035] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e1b6dbbd564152b350307b57f5fc06" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:16.543813 11002 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:44:16.544070 11002 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:44:16.547060 11002 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b7a3845d3ed41028340970bdf39db98: Sending DeleteTablet for 3 replicas of tablet 66ba967ba60049ada25cb4cff71b4b82
I20260621 02:44:16.548105 11276 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ba967ba60049ada25cb4cff71b4b82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:16 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:48044
I20260621 02:44:16.548106 11416 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ba967ba60049ada25cb4cff71b4b82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:16 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:55386
I20260621 02:44:16.548666 11138 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ba967ba60049ada25cb4cff71b4b82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:16 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:53350
I20260621 02:44:16.560209 11563 tablet_replica.cc:330] T 66ba967ba60049ada25cb4cff71b4b82 P a5e1b6dbbd564152b350307b57f5fc06: stopping tablet replica
I20260621 02:44:16.560489 11563 raft_consensus.cc:2229] T 66ba967ba60049ada25cb4cff71b4b82 P a5e1b6dbbd564152b350307b57f5fc06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:44:16.560738 11563 raft_consensus.cc:2258] T 66ba967ba60049ada25cb4cff71b4b82 P a5e1b6dbbd564152b350307b57f5fc06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:16.562827 11563 ts_tablet_manager.cc:1905] T 66ba967ba60049ada25cb4cff71b4b82 P a5e1b6dbbd564152b350307b57f5fc06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:16.563823 11564 tablet_replica.cc:330] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda: stopping tablet replica
I20260621 02:44:16.564087 11564 raft_consensus.cc:2229] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:44:16.564662 11564 raft_consensus.cc:2258] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:16.565801 11564 ts_tablet_manager.cc:1905] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:16.566761 11562 tablet_replica.cc:330] T 66ba967ba60049ada25cb4cff71b4b82 P cca8c39ad4db4bc59c244e008a3dacea: stopping tablet replica
I20260621 02:44:16.566954 11562 raft_consensus.cc:2229] T 66ba967ba60049ada25cb4cff71b4b82 P cca8c39ad4db4bc59c244e008a3dacea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:44:16.567158 11562 raft_consensus.cc:2258] T 66ba967ba60049ada25cb4cff71b4b82 P cca8c39ad4db4bc59c244e008a3dacea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:16.568233 11562 ts_tablet_manager.cc:1905] T 66ba967ba60049ada25cb4cff71b4b82 P cca8c39ad4db4bc59c244e008a3dacea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:16.570603 11563 ts_tablet_manager.cc:1918] T 66ba967ba60049ada25cb4cff71b4b82 P a5e1b6dbbd564152b350307b57f5fc06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:44:16.570752 11563 log.cc:1196] T 66ba967ba60049ada25cb4cff71b4b82 P a5e1b6dbbd564152b350307b57f5fc06: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/66ba967ba60049ada25cb4cff71b4b82
I20260621 02:44:16.571174 11563 ts_tablet_manager.cc:1939] T 66ba967ba60049ada25cb4cff71b4b82 P a5e1b6dbbd564152b350307b57f5fc06: Deleting consensus metadata
I20260621 02:44:16.572569 10986 catalog_manager.cc:4841] TS a5e1b6dbbd564152b350307b57f5fc06 (127.9.166.2:42511): tablet 66ba967ba60049ada25cb4cff71b4b82 (table test-table [id=be6e02ff1c954c28874f6668be537c89]) successfully deleted
I20260621 02:44:16.574311 11562 ts_tablet_manager.cc:1918] T 66ba967ba60049ada25cb4cff71b4b82 P cca8c39ad4db4bc59c244e008a3dacea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:44:16.574375 11564 ts_tablet_manager.cc:1918] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:44:16.574450 11562 log.cc:1196] T 66ba967ba60049ada25cb4cff71b4b82 P cca8c39ad4db4bc59c244e008a3dacea: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/66ba967ba60049ada25cb4cff71b4b82
I20260621 02:44:16.574505 11564 log.cc:1196] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/66ba967ba60049ada25cb4cff71b4b82
I20260621 02:44:16.574828 11562 ts_tablet_manager.cc:1939] T 66ba967ba60049ada25cb4cff71b4b82 P cca8c39ad4db4bc59c244e008a3dacea: Deleting consensus metadata
I20260621 02:44:16.574970 11564 ts_tablet_manager.cc:1939] T 66ba967ba60049ada25cb4cff71b4b82 P de7c7c91e26c44208e5999fa2738dfda: Deleting consensus metadata
I20260621 02:44:16.575834 10989 catalog_manager.cc:4841] TS cca8c39ad4db4bc59c244e008a3dacea (127.9.166.3:40503): tablet 66ba967ba60049ada25cb4cff71b4b82 (table test-table [id=be6e02ff1c954c28874f6668be537c89]) successfully deleted
I20260621 02:44:16.576138 10987 catalog_manager.cc:4841] TS de7c7c91e26c44208e5999fa2738dfda (127.9.166.1:37681): tablet 66ba967ba60049ada25cb4cff71b4b82 (table test-table [id=be6e02ff1c954c28874f6668be537c89]) successfully deleted
Jun 21 02:44:16 dist-test-slave-hvgn krb5kdc[10957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009855, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:44:16.585719 11501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:16.560035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53386 (local address 127.9.166.1:37681)
0621 02:44:16.560204 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:16.560210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:16.566427 (+ 6217us) server_negotiation.cc:407] Connection header received
0621 02:44:16.567623 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:16.567631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:16.567705 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:16.567789 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:16.569847 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.570954 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.572899 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.573543 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.573631 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:16.578831 (+ 5200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:16.578844 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:16.578863 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:16.578921 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:16.582803 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:16.583475 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:16.583485 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:16.583489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:16.583673 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:16.584025 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:16.584030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:16.584033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:16.584482 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:16.584805 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:16.585105 (+ 300us) server_negotiation.cc:299] Negotiation successful
0621 02:44:16.585348 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260621 02:44:16.586408 11203 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53386
I20260621 02:44:16.602514 11545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:16.589446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41330 (local address 127.9.166.62:34155)
0621 02:44:16.589684 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:16.589690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:16.589923 (+ 233us) server_negotiation.cc:407] Connection header received
0621 02:44:16.590111 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:16.590115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:16.590176 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:16.590249 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:16.591874 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.592592 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.594429 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.595004 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.595148 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:16.597361 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:16.597367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:16.597372 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:16.597418 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:16.600040 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:16.600671 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:16.600676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:16.600679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:16.600802 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:16.601161 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:16.601168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:16.601171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:16.601431 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:16.601687 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:16.601982 (+ 295us) server_negotiation.cc:299] Negotiation successful
0621 02:44:16.602199 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260621 02:44:16.603051 11002 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41330
I20260621 02:44:16.605178 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 11073
I20260621 02:44:16.619488 11501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:16.606301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53392 (local address 127.9.166.1:37681)
0621 02:44:16.606494 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:16.606500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:16.606518 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:44:16.606686 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:16.606691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:16.606762 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:16.606834 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:16.608129 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.609249 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.610365 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.611039 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.611229 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:16.613345 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:16.613352 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:16.613357 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:16.613397 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:16.617054 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:16.617606 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:16.617612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:16.617616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:16.617775 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:16.618080 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:16.618085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:16.618089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:16.618395 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:16.618695 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:16.618844 (+ 149us) server_negotiation.cc:299] Negotiation successful
0621 02:44:16.619061 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260621 02:44:16.620132 11203 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53392
W20260621 02:44:16.620636 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 11073: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:44:16.620827 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 11073
I20260621 02:44:16.640180 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 11212
Jun 21 02:44:16 dist-test-slave-hvgn krb5kdc[10957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009855, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.166.2@KRBTEST.COM
I20260621 02:44:16.655478 11570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:16.640967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48058 (local address 127.9.166.2:42511)
0621 02:44:16.641364 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:16.641369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:16.641385 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:44:16.641439 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:16.641442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:16.641508 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:16.641597 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:16.642723 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.643510 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.644662 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.645201 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.645306 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:16.649505 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:16.649521 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:16.649536 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:16.649573 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:16.652575 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:16.653272 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:16.653288 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:16.653290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:16.653418 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:16.653798 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:16.653803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:16.653808 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:16.654158 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:16.654489 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:16.654821 (+ 332us) server_negotiation.cc:299] Negotiation successful
0621 02:44:16.655061 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":197,"threads_started":1}
W20260621 02:44:16.656380 11342 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48058
W20260621 02:44:16.657133 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 11212: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:44:16.657284 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 11212
I20260621 02:44:16.677873 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 11351
Jun 21 02:44:16 dist-test-slave-hvgn krb5kdc[10957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009855, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.166.3@KRBTEST.COM
I20260621 02:44:16.692608 11571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:16.678693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55388 (local address 127.9.166.3:40503)
0621 02:44:16.679194 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:16.679198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:16.679216 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:44:16.679296 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:16.679299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:16.679361 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:16.679530 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:16.680695 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.681414 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.682539 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:16.683025 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:16.683129 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:16.687210 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:16.687225 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:16.687243 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:16.687316 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:16.690170 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:16.690754 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:16.690761 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:16.690764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:16.690887 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:16.691175 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:16.691178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:16.691180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:16.691728 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:16.691946 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:16.692181 (+ 235us) server_negotiation.cc:299] Negotiation successful
0621 02:44:16.692346 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":300,"threads_started":1}
W20260621 02:44:16.693250 11481 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55388
W20260621 02:44:16.693900 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 11351: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:44:16.694054 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 11351
I20260621 02:44:16.710796 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 10970
I20260621 02:44:16.711989 11042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:16.826365 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 10970
2026-06-21T02:44:16Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4630 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:44:16 dist-test-slave-hvgn krb5kdc[11575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:44:16 dist-test-slave-hvgn krb5kdc[11575](info): set up 2 sockets
Jun 21 02:44:16 dist-test-slave-hvgn krb5kdc[11575](info): commencing operation
krb5kdc: starting...
W20260621 02:44:17.923583 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:44:18 dist-test-slave-hvgn krb5kdc[11575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:44:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:44:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:44:18.102972 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:37717
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:33995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:37717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:18.418687 11588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:18.419020 11588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:18.419118 11588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:18.426272 11588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:44:18.426379 11588 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 02:44:18.426420 11588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:18.426478 11588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:18.426528 11588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:44:18.426581 11588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:44:18.436523 11588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:33995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:37717
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:37717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:18.437770 11588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:18.439162 11588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:18.451135 11593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:18.451234 11594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:18.451176 11596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:18.452395 11588 server_base.cc:1034] running on GCE node
I20260621 02:44:18.453167 11588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:18.454571 11588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:18.455837 11588 hybrid_clock.cc:648] HybridClock initialized: now 1782009858455774 us; error 56 us; skew 500 ppm
I20260621 02:44:18.456187 11588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:18 dist-test-slave-hvgn krb5kdc[11575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009858, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:18.459537 11588 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:44:18.461023 11588 webserver.cc:460] Webserver started at http://127.9.166.62:35343/ using document root <none> and password file <none>
I20260621 02:44:18.461783 11588 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:18.461905 11588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:18.462252 11588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:18.465380 11588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "c038bcc03ac44463a728929ccc853658"
format_stamp: "Formatted at 2026-06-21 02:44:18 on dist-test-slave-hvgn"
server_key: "9c71e4c9e0d4cb41e4919b84b82f5241"
server_key_iv: "f0399f5d691e0db73913d127e83199c5"
server_key_version: "encryptionkey@0"
I20260621 02:44:18.466190 11588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "c038bcc03ac44463a728929ccc853658"
format_stamp: "Formatted at 2026-06-21 02:44:18 on dist-test-slave-hvgn"
server_key: "9c71e4c9e0d4cb41e4919b84b82f5241"
server_key_iv: "f0399f5d691e0db73913d127e83199c5"
server_key_version: "encryptionkey@0"
I20260621 02:44:18.471589 11588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 02:44:18.475189 11603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:18.476614 11588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:44:18.476840 11588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "c038bcc03ac44463a728929ccc853658"
format_stamp: "Formatted at 2026-06-21 02:44:18 on dist-test-slave-hvgn"
server_key: "9c71e4c9e0d4cb41e4919b84b82f5241"
server_key_iv: "f0399f5d691e0db73913d127e83199c5"
server_key_version: "encryptionkey@0"
I20260621 02:44:18.477034 11588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:18.501991 11588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:18.506212 11588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:18.506501 11588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:18.529165 11588 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:37717
I20260621 02:44:18.529184 11664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:37717 every 8 connection(s)
I20260621 02:44:18.530575 11588 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:44:18.534302 11665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:18.540560 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 11588
I20260621 02:44:18.540787 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:44:18.541610 9880 external_mini_cluster.cc:1428] Setting key b65bcee3cafee16bcebbb1ae9205786b
I20260621 02:44:18.545316 11665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658: Bootstrap starting.
I20260621 02:44:18.548830 11665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:18.550071 11665 log.cc:824] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:18.553332 11665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658: No bootstrap required, opened a new log
Jun 21 02:44:18 dist-test-slave-hvgn krb5kdc[11575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:18.562422 11665 raft_consensus.cc:348] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c038bcc03ac44463a728929ccc853658" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37717 } }
I20260621 02:44:18.562778 11665 raft_consensus.cc:374] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:18.562924 11665 raft_consensus.cc:729] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c038bcc03ac44463a728929ccc853658, State: Initialized, Role: FOLLOWER
I20260621 02:44:18.563625 11665 consensus_queue.cc:260] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c038bcc03ac44463a728929ccc853658" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37717 } }
I20260621 02:44:18.563861 11665 raft_consensus.cc:388] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:44:18.563998 11665 raft_consensus.cc:482] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:44:18.564186 11665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:18.567564 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:18.543819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40598 (local address 127.9.166.62:37717)
0621 02:44:18.544665 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:18.544678 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:18.544719 (+ 41us) server_negotiation.cc:407] Connection header received
0621 02:44:18.546629 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:18.546656 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:18.547073 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:18.547468 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:18.549311 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:18.551313 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:18.553027 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:18.553839 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:18.553981 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:18.558597 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:18.558604 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:18.558621 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:18.558655 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:18.561989 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:18.562666 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:18.562694 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:18.562703 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:18.562927 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:18.563345 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:18.563349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:18.563353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:18.563976 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:18.564309 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:18.565540 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0621 02:44:18.566044 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":406,"threads_started":1}
I20260621 02:44:18.567534 11665 raft_consensus.cc:504] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c038bcc03ac44463a728929ccc853658" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37717 } }
I20260621 02:44:18.568068 11665 leader_election.cc:302] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c038bcc03ac44463a728929ccc853658; no voters:
I20260621 02:44:18.568725 11665 leader_election.cc:288] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:44:18.568854 11670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:18.570031 11670 raft_consensus.cc:686] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658 [term 1 LEADER]: Becoming Leader. State: Replica: c038bcc03ac44463a728929ccc853658, State: Running, Role: LEADER
I20260621 02:44:18.570688 11670 consensus_queue.cc:237] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c038bcc03ac44463a728929ccc853658" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37717 } }
I20260621 02:44:18.570984 11665 sys_catalog.cc:564] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:44:18.573451 11672 sys_catalog.cc:455] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c038bcc03ac44463a728929ccc853658. Latest consensus state: current_term: 1 leader_uuid: "c038bcc03ac44463a728929ccc853658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c038bcc03ac44463a728929ccc853658" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37717 } } }
I20260621 02:44:18.573760 11672 sys_catalog.cc:458] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:18.573562 11671 sys_catalog.cc:455] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c038bcc03ac44463a728929ccc853658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c038bcc03ac44463a728929ccc853658" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37717 } } }
I20260621 02:44:18.573927 11671 sys_catalog.cc:458] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:18.574671 11676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:44:18.580994 11676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:44:18.591241 11676 catalog_manager.cc:1347] Generated new cluster ID: 35a085ce87dc402c83de4b61e6c60ce5
I20260621 02:44:18.591368 11676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:44:18.604506 11676 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:44:18.606294 11676 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:44:18.622547 11676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658: Generated new TSK 0
I20260621 02:44:18.623596 11676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
I20260621 02:44:18.674523 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:37717
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:33995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:19.010509 11691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:19.010758 11691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:19.010931 11691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:19.017956 11691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:19.018055 11691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:19.018213 11691 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 02:44:19.018270 11691 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:44:19.018368 11691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:44:19.027630 11691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:33995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.166.62:37717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:19.028839 11691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:19.030158 11691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:19.043035 11696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:19.043078 11697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:19.043253 11699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:19.044324 11691 server_base.cc:1034] running on GCE node
I20260621 02:44:19.044903 11691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:19.045778 11691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:19.047027 11691 hybrid_clock.cc:648] HybridClock initialized: now 1782009859046974 us; error 47 us; skew 500 ppm
I20260621 02:44:19.047330 11691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:19 dist-test-slave-hvgn krb5kdc[11575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009859, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:19.050355 11691 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:44:19.052052 11691 webserver.cc:460] Webserver started at http://127.9.166.1:37145/ using document root <none> and password file <none>
I20260621 02:44:19.052711 11691 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:19.052853 11691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:19.053165 11691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:19.056061 11691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "1ea7d2e312c54d69a527f948299ed79b"
format_stamp: "Formatted at 2026-06-21 02:44:19 on dist-test-slave-hvgn"
server_key: "b92616b2b143ad25a758e0245bbead9f"
server_key_iv: "9adbfa2a3732b2271d58fd04d2012eb9"
server_key_version: "encryptionkey@0"
I20260621 02:44:19.056789 11691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "1ea7d2e312c54d69a527f948299ed79b"
format_stamp: "Formatted at 2026-06-21 02:44:19 on dist-test-slave-hvgn"
server_key: "b92616b2b143ad25a758e0245bbead9f"
server_key_iv: "9adbfa2a3732b2271d58fd04d2012eb9"
server_key_version: "encryptionkey@0"
I20260621 02:44:19.061707 11691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:44:19.065053 11706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:19.066458 11691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:44:19.066583 11691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "1ea7d2e312c54d69a527f948299ed79b"
format_stamp: "Formatted at 2026-06-21 02:44:19 on dist-test-slave-hvgn"
server_key: "b92616b2b143ad25a758e0245bbead9f"
server_key_iv: "9adbfa2a3732b2271d58fd04d2012eb9"
server_key_version: "encryptionkey@0"
I20260621 02:44:19.066732 11691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:19.080680 11691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:19.084515 11691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:19.084736 11691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:19.087144 11691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:19.087246 11691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:19.087433 11691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:19.087527 11691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:44:19 dist-test-slave-hvgn krb5kdc[11575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009859, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:19.117110 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:19.087511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:45939 (local address 127.9.166.62:37717)
0621 02:44:19.093952 (+ 6441us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:19.093960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:19.093993 (+ 33us) server_negotiation.cc:407] Connection header received
0621 02:44:19.096382 (+ 2389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:19.096387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:19.096452 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:19.096550 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:19.098871 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:19.100171 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:19.103171 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:19.103882 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:19.104032 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:19.108872 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:19.108877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:19.108881 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:19.108918 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:19.112842 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:19.113803 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:19.113811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:19.113817 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:19.113977 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:19.114508 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:19.114516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:19.114522 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:19.114977 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:19.115149 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:19.116266 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0621 02:44:19.116568 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6305,"thread_start_us":6218,"threads_started":1}
I20260621 02:44:19.117707 11716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:19.088276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37717 (local address 127.9.166.1:45939)
0621 02:44:19.093137 (+ 4861us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:19.093205 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:19.096061 (+ 2856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:19.096860 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:19.096871 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:19.097653 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:19.098652 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:19.098685 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:19.100350 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:19.100358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:19.103013 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:19.103031 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:19.104154 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:19.104164 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:19.104379 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:19.105234 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:19.105272 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:19.108643 (+ 3371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:19.113032 (+ 4389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:19.113043 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:19.113063 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:19.113668 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:19.114129 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:19.114134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:19.114136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:19.114337 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:19.115273 (+ 936us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:19.115284 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:19.115743 (+ 459us) client_negotiation.cc:769] Sending connection context
0621 02:44:19.116147 (+ 404us) client_negotiation.cc:240] Negotiation successful
0621 02:44:19.116536 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4443,"thread_start_us":4182,"threads_started":1}
I20260621 02:44:19.136437 11691 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:42573
I20260621 02:44:19.136543 11826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:42573 every 8 connection(s)
I20260621 02:44:19.137714 11691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:44:19.138796 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:19.125002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:57211 (local address 127.9.166.62:37717)
0621 02:44:19.125235 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:19.125242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:19.126332 (+ 1090us) server_negotiation.cc:407] Connection header received
0621 02:44:19.126553 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:19.126558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:19.126640 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:19.126739 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:19.127741 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:19.128774 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:19.130606 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:19.131185 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:19.131304 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:19.133417 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:19.133423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:19.133428 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:19.133467 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:19.136094 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:19.136913 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:19.136920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:19.136924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:19.137052 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:19.137415 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:19.137420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:19.137424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:19.137676 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:19.137836 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:19.138233 (+ 397us) server_negotiation.cc:299] Negotiation successful
0621 02:44:19.138445 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:44:19.138937 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:19.125393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37717 (local address 127.9.166.1:57211)
0621 02:44:19.126200 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:19.126229 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:19.126403 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:19.126778 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:19.126782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:19.127192 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:19.127603 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:19.127613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:19.128907 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:19.128917 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:19.130467 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:19.130479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:19.131350 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:19.131359 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:19.131551 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:19.132220 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:19.132249 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:19.133247 (+ 998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:19.136227 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:19.136232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:19.136236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:19.136803 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:19.137160 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:19.137167 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:19.137174 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:19.137304 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:19.137802 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:19.137807 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:19.137986 (+ 179us) client_negotiation.cc:769] Sending connection context
0621 02:44:19.138126 (+ 140us) client_negotiation.cc:240] Negotiation successful
0621 02:44:19.138464 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":541,"threads_started":1}
I20260621 02:44:19.140168 11827 heartbeater.cc:347] Connected to a master server at 127.9.166.62:37717
I20260621 02:44:19.140482 11827 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:19.141539 11827 heartbeater.cc:511] Master 127.9.166.62:37717 requested a full tablet report, sending...
I20260621 02:44:19.143507 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 11691
I20260621 02:44:19.143714 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:44:19.143980 11620 ts_manager.cc:194] Registered new tserver with Master: 1ea7d2e312c54d69a527f948299ed79b (127.9.166.1:42573)
I20260621 02:44:19.144294 9880 external_mini_cluster.cc:1428] Setting key 930c3c989b69870f8d72ca0e719487b5
I20260621 02:44:19.145622 11620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:45939
WARNING: no policy specified for kudu/127.9.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
I20260621 02:44:19.183005 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:37717
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:33995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:19.502207 11831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:19.502512 11831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:19.502668 11831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:19.509694 11831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:19.509831 11831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:19.509982 11831 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 02:44:19.510077 11831 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:44:19.510207 11831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:44:19.519517 11831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:33995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.166.62:37717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:19.520891 11831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:19.522269 11831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:19.535279 11836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:19.535320 11839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:19.535347 11837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:19.537004 11831 server_base.cc:1034] running on GCE node
I20260621 02:44:19.537678 11831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:19.538703 11831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:19.539944 11831 hybrid_clock.cc:648] HybridClock initialized: now 1782009859539890 us; error 44 us; skew 500 ppm
I20260621 02:44:19.540297 11831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:19 dist-test-slave-hvgn krb5kdc[11575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009859, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:19.544095 11831 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:44:19.545578 11831 webserver.cc:460] Webserver started at http://127.9.166.2:42749/ using document root <none> and password file <none>
I20260621 02:44:19.546289 11831 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:19.546416 11831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:19.546736 11831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:19.549685 11831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "fbc0f56a333b442d8f50f6a5eb21d138"
format_stamp: "Formatted at 2026-06-21 02:44:19 on dist-test-slave-hvgn"
server_key: "0dfaddba206e4fb1a3c5611af177db9e"
server_key_iv: "663e489d9b4387c90733b326c9223c00"
server_key_version: "encryptionkey@0"
I20260621 02:44:19.550427 11831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "fbc0f56a333b442d8f50f6a5eb21d138"
format_stamp: "Formatted at 2026-06-21 02:44:19 on dist-test-slave-hvgn"
server_key: "0dfaddba206e4fb1a3c5611af177db9e"
server_key_iv: "663e489d9b4387c90733b326c9223c00"
server_key_version: "encryptionkey@0"
I20260621 02:44:19.555568 11831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 02:44:19.559002 11846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:19.560397 11831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:44:19.560520 11831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "fbc0f56a333b442d8f50f6a5eb21d138"
format_stamp: "Formatted at 2026-06-21 02:44:19 on dist-test-slave-hvgn"
server_key: "0dfaddba206e4fb1a3c5611af177db9e"
server_key_iv: "663e489d9b4387c90733b326c9223c00"
server_key_version: "encryptionkey@0"
I20260621 02:44:19.560673 11831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:19.573632 11831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:19.577739 11831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:19.577975 11831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:19.580741 11831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:19.580888 11831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:19.581010 11831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:19.581102 11831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:44:19 dist-test-slave-hvgn krb5kdc[11575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009859, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:19.606869 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:19.582198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37717 (local address 127.9.166.2:34817)
0621 02:44:19.582980 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:19.583021 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:19.585208 (+ 2187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:19.586292 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:19.586306 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:19.587364 (+ 1058us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:19.588554 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:19.588626 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:19.590157 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:19.590161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:19.592234 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:19.592254 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:19.593597 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:19.593602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:19.593798 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:19.594603 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:19.594626 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:19.597933 (+ 3307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:19.601965 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:19.601976 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:19.601995 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:19.602654 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:19.603052 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:19.603061 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:19.603064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:19.603277 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:19.603949 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:19.603957 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:19.604353 (+ 396us) client_negotiation.cc:769] Sending connection context
0621 02:44:19.604721 (+ 368us) client_negotiation.cc:240] Negotiation successful
0621 02:44:19.605315 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":362,"threads_started":1}
I20260621 02:44:19.607661 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:19.581448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:34817 (local address 127.9.166.62:37717)
0621 02:44:19.581637 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:19.581646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:19.583182 (+ 1536us) server_negotiation.cc:407] Connection header received
0621 02:44:19.585538 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:19.585544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:19.585678 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:19.585813 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:19.588834 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:19.589944 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:19.592421 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:19.593321 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:19.593490 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:19.598241 (+ 4751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:19.598250 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:19.598255 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:19.598303 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:19.601761 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:19.602820 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:19.602824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:19.602826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:19.602940 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:19.603427 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:19.603434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:19.603439 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:19.603819 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:19.603972 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:19.607027 (+ 3055us) server_negotiation.cc:299] Negotiation successful
0621 02:44:19.607273 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 02:44:19.627035 11831 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:34747
I20260621 02:44:19.627081 11965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:34747 every 8 connection(s)
I20260621 02:44:19.628358 11831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:44:19.628368 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:19.613107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:50001 (local address 127.9.166.62:37717)
0621 02:44:19.613293 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:19.613298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:19.614316 (+ 1018us) server_negotiation.cc:407] Connection header received
0621 02:44:19.614537 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:19.614542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:19.614614 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:19.614713 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:19.615653 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:19.616911 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:19.618559 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:19.619193 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:19.619334 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:19.621642 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:19.621651 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:19.621659 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:19.621706 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:19.625479 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:19.626373 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:19.626377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:19.626379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:19.626493 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:19.626870 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:19.626877 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:19.626882 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:19.627250 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:19.627438 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:19.627804 (+ 366us) server_negotiation.cc:299] Negotiation successful
0621 02:44:19.628015 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 02:44:19.628491 11932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:19.613352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37717 (local address 127.9.166.2:50001)
0621 02:44:19.614195 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:19.614218 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:19.614374 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:19.614740 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:19.614745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:19.615152 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:19.615493 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:19.615509 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:19.617079 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:19.617133 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:19.618378 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:19.618396 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:19.619373 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:19.619379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:19.619576 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:19.620254 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:19.620283 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:19.621445 (+ 1162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:19.625630 (+ 4185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:19.625633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:19.625636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:19.626205 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:19.626625 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:19.626630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:19.626634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:19.626762 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:19.627372 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:19.627377 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:19.627574 (+ 197us) client_negotiation.cc:769] Sending connection context
0621 02:44:19.627719 (+ 145us) client_negotiation.cc:240] Negotiation successful
0621 02:44:19.628022 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":536,"threads_started":1}
I20260621 02:44:19.629455 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 11831
I20260621 02:44:19.629691 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:44:19.630271 11966 heartbeater.cc:347] Connected to a master server at 127.9.166.62:37717
I20260621 02:44:19.630373 9880 external_mini_cluster.cc:1428] Setting key 27d0f7900a44659b89ef4b30db5df1b4
I20260621 02:44:19.630513 11966 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:19.631258 11966 heartbeater.cc:511] Master 127.9.166.62:37717 requested a full tablet report, sending...
I20260621 02:44:19.634356 11619 ts_manager.cc:194] Registered new tserver with Master: fbc0f56a333b442d8f50f6a5eb21d138 (127.9.166.2:34747)
I20260621 02:44:19.635228 11619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:34817
WARNING: no policy specified for kudu/127.9.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
I20260621 02:44:19.673840 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:37717
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:33995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:19.991842 11970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:19.992141 11970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:19.992242 11970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:19.999399 11970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:19.999495 11970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:19.999576 11970 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260621 02:44:19.999629 11970 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:44:19.999688 11970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:44:20.009322 11970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:33995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.166.62:37717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:20.010454 11970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:20.011863 11970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:20.024391 11976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:20.024575 11975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:20.024330 11978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:20.025434 11970 server_base.cc:1034] running on GCE node
I20260621 02:44:20.025939 11970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:20.026746 11970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:20.027969 11970 hybrid_clock.cc:648] HybridClock initialized: now 1782009860027935 us; error 46 us; skew 500 ppm
I20260621 02:44:20.028313 11970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:20 dist-test-slave-hvgn krb5kdc[11575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009860, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:20.036734 11970 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:44:20.038166 11970 webserver.cc:460] Webserver started at http://127.9.166.3:46519/ using document root <none> and password file <none>
I20260621 02:44:20.038851 11970 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:20.038993 11970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:20.039310 11970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:20.042078 11970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "fce76e8380224ee5b83154d8c9ba82dc"
format_stamp: "Formatted at 2026-06-21 02:44:20 on dist-test-slave-hvgn"
server_key: "3679a7aaa9d51241e62e21f646781dc1"
server_key_iv: "8e07671abe3f7af3cde24c975e68a18e"
server_key_version: "encryptionkey@0"
I20260621 02:44:20.042811 11970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "fce76e8380224ee5b83154d8c9ba82dc"
format_stamp: "Formatted at 2026-06-21 02:44:20 on dist-test-slave-hvgn"
server_key: "3679a7aaa9d51241e62e21f646781dc1"
server_key_iv: "8e07671abe3f7af3cde24c975e68a18e"
server_key_version: "encryptionkey@0"
I20260621 02:44:20.047770 11970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260621 02:44:20.050945 11985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:20.052205 11970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:44:20.052374 11970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "fce76e8380224ee5b83154d8c9ba82dc"
format_stamp: "Formatted at 2026-06-21 02:44:20 on dist-test-slave-hvgn"
server_key: "3679a7aaa9d51241e62e21f646781dc1"
server_key_iv: "8e07671abe3f7af3cde24c975e68a18e"
server_key_version: "encryptionkey@0"
I20260621 02:44:20.052563 11970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:20.083184 11970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:20.087194 11970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:20.087548 11970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:20.090204 11970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:20.090332 11970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:20.090481 11970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:20.090574 11970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:44:20 dist-test-slave-hvgn krb5kdc[11575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009860, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:20.118397 11994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.091659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37717 (local address 127.9.166.3:38513)
0621 02:44:20.092557 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:20.092616 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:20.095346 (+ 2730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:20.096263 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:20.096274 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:20.097220 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:20.098246 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.098276 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.099789 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.099801 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.102588 (+ 2787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.102626 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.103621 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.103630 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.104021 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.104970 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:20.105006 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:20.108571 (+ 3565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:20.113211 (+ 4640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:20.113226 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:20.113243 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:20.113894 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:20.114372 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:20.114384 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:20.114390 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:20.114810 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:20.115451 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:20.115461 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:20.115843 (+ 382us) client_negotiation.cc:769] Sending connection context
0621 02:44:20.116245 (+ 402us) client_negotiation.cc:240] Negotiation successful
0621 02:44:20.116826 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":340,"threads_started":1}
I20260621 02:44:20.119460 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.091253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:38513 (local address 127.9.166.62:37717)
0621 02:44:20.091596 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:20.091605 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:20.092759 (+ 1154us) server_negotiation.cc:407] Connection header received
0621 02:44:20.095714 (+ 2955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:20.095723 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:20.095829 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:20.095950 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:20.098531 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.099604 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.102791 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.103323 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.103510 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.108836 (+ 5326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:20.108845 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:20.108852 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:20.108906 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:20.113055 (+ 4149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:20.114034 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:20.114041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:20.114045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:20.114226 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:20.114954 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:20.114961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:20.114966 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:20.115279 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:20.115450 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:20.118957 (+ 3507us) server_negotiation.cc:299] Negotiation successful
0621 02:44:20.119156 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260621 02:44:20.141583 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.125020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37717 (local address 127.9.166.3:33919)
0621 02:44:20.126366 (+ 1346us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:20.126387 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:20.126553 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:20.126967 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:20.126975 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:20.127686 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:20.128091 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.128109 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.129480 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.129489 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.131760 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.131784 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.132552 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.132565 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.132789 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.133562 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:20.133596 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:20.134914 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:20.138519 (+ 3605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:20.138527 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:20.138534 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:20.139216 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:20.139677 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:20.139682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:20.139686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:20.139838 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:20.140466 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:20.140471 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:20.140656 (+ 185us) client_negotiation.cc:769] Sending connection context
0621 02:44:20.140824 (+ 168us) client_negotiation.cc:240] Negotiation successful
0621 02:44:20.141136 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1122,"thread_start_us":940,"threads_started":1}
I20260621 02:44:20.142732 11970 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:41199
I20260621 02:44:20.142751 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.125014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:33919 (local address 127.9.166.62:37717)
0621 02:44:20.125271 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:20.125280 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:20.126491 (+ 1211us) server_negotiation.cc:407] Connection header received
0621 02:44:20.126705 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:20.126713 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:20.126796 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:20.126907 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:20.128237 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.129290 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.131921 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.132352 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.132444 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.135142 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:20.135147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:20.135152 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:20.135187 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:20.138384 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:20.139372 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:20.139377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:20.139381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:20.139548 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:20.139954 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:20.139959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:20.139963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:20.140261 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:20.140403 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:20.142335 (+ 1932us) server_negotiation.cc:299] Negotiation successful
0621 02:44:20.142539 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:44:20.142784 12104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:41199 every 8 connection(s)
I20260621 02:44:20.144028 11970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:44:20.146256 12105 heartbeater.cc:347] Connected to a master server at 127.9.166.62:37717
I20260621 02:44:20.146520 12105 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:20.147104 12105 heartbeater.cc:511] Master 127.9.166.62:37717 requested a full tablet report, sending...
I20260621 02:44:20.148278 11827 heartbeater.cc:503] Master 127.9.166.62:37717 was elected leader, sending a full tablet report...
I20260621 02:44:20.148417 11619 ts_manager.cc:194] Registered new tserver with Master: fce76e8380224ee5b83154d8c9ba82dc (127.9.166.3:41199)
I20260621 02:44:20.149276 11619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:38513
I20260621 02:44:20.151190 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 11970
I20260621 02:44:20.151350 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:44:20.151791 9880 external_mini_cluster.cc:1428] Setting key 1c538d8083ff386bcc040bdc6c5237eb
I20260621 02:44:20.154338 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 02:44:20 dist-test-slave-hvgn krb5kdc[11575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 02:44:20 dist-test-slave-hvgn krb5kdc[11575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:20.198989 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.185612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40604 (local address 127.9.166.62:37717)
0621 02:44:20.185838 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:20.185845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:20.185966 (+ 121us) server_negotiation.cc:407] Connection header received
0621 02:44:20.186183 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:20.186187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:20.186297 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:20.186409 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:20.187787 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.188512 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.189755 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.190141 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.190280 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.194142 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:20.194147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:20.194150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:20.194194 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:20.196810 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:20.197383 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:20.197387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:20.197389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:20.197513 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:20.197841 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:20.197844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:20.197846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:20.198097 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:20.198272 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:20.198529 (+ 257us) server_negotiation.cc:299] Negotiation successful
0621 02:44:20.198702 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:44:20.202872 11619 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:44:20.209591 11619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:44:20.232178 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.221538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:34747 (local address 127.0.0.1:38932)
0621 02:44:20.222478 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:20.222524 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:20.222733 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:20.224774 (+ 2041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:20.224782 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:20.224823 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:20.225196 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.225205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.226757 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.226763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.228434 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.228446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.231125 (+ 2679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.231136 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.231309 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.231357 (+ 48us) client_negotiation.cc:769] Sending connection context
0621 02:44:20.231531 (+ 174us) client_negotiation.cc:240] Negotiation successful
0621 02:44:20.231724 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":3,"spinlock_wait_cycles":14464,"thread_start_us":762,"threads_started":1}
I20260621 02:44:20.233345 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.221670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.9.166.2:34747)
0621 02:44:20.224122 (+ 2452us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:20.224131 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:20.224159 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:44:20.224253 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:20.224259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:20.224518 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:20.225372 (+ 854us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:20.225513 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.226613 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.228583 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.230907 (+ 2324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.232638 (+ 1731us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.232856 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:20.232936 (+ 80us) server_negotiation.cc:299] Negotiation successful
0621 02:44:20.233117 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2302,"thread_start_us":2192,"threads_started":1}
I20260621 02:44:20.236649 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.221654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:41199 (local address 127.0.0.1:44034)
0621 02:44:20.223440 (+ 1786us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:20.223461 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:20.223585 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:20.229729 (+ 6144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:20.229737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:20.229776 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:20.230254 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.230266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.232049 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.232057 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.233920 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.233934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.235728 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.235734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.235978 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.236010 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 02:44:20.236110 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 02:44:20.236244 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1617,"mutex_wait_us":37,"thread_start_us":1427,"threads_started":1}
I20260621 02:44:20.236656 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.222129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44034 (local address 127.9.166.3:41199)
0621 02:44:20.229104 (+ 6975us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:20.229113 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:20.229144 (+ 31us) server_negotiation.cc:407] Connection header received
0621 02:44:20.229234 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:20.229242 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:20.229514 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:20.229712 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:20.230461 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.231929 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.234106 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.235533 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.235661 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.235875 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:20.236180 (+ 305us) server_negotiation.cc:299] Negotiation successful
0621 02:44:20.236344 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6805,"thread_start_us":6622,"threads_started":1}
I20260621 02:44:20.237237 11896 tablet_service.cc:1467] Processing CreateTablet for tablet 2d1869fab6f64b8ba73c1e4786ac9b6e (DEFAULT_TABLE table=test-table [id=076ad3ab5b9046b2bd88373d4be7b7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:20.238651 11896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d1869fab6f64b8ba73c1e4786ac9b6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:20.239287 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.223740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.9.166.1:42573)
0621 02:44:20.230379 (+ 6639us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:20.230389 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:20.230417 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:44:20.230506 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:20.230514 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:20.230726 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:20.230902 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:20.231562 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.232987 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.236661 (+ 3674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.238054 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.238189 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.238387 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:20.238794 (+ 407us) server_negotiation.cc:299] Negotiation successful
0621 02:44:20.238978 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6476,"thread_start_us":6344,"threads_started":1}
I20260621 02:44:20.239835 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.224169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:42573 (local address 127.0.0.1:60626)
0621 02:44:20.224763 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:20.224779 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:20.224875 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:20.230921 (+ 6046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:20.230927 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:20.230954 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:20.231364 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.231375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.233125 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.233131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.236474 (+ 3343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.236488 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.238254 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.238261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.238437 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.238465 (+ 28us) client_negotiation.cc:769] Sending connection context
0621 02:44:20.238557 (+ 92us) client_negotiation.cc:240] Negotiation successful
0621 02:44:20.238681 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":33,"spinlock_wait_cycles":768,"thread_start_us":312,"threads_started":1}
I20260621 02:44:20.240806 12035 tablet_service.cc:1467] Processing CreateTablet for tablet 2d1869fab6f64b8ba73c1e4786ac9b6e (DEFAULT_TABLE table=test-table [id=076ad3ab5b9046b2bd88373d4be7b7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:20.242236 12035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d1869fab6f64b8ba73c1e4786ac9b6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:20.247242 11757 tablet_service.cc:1467] Processing CreateTablet for tablet 2d1869fab6f64b8ba73c1e4786ac9b6e (DEFAULT_TABLE table=test-table [id=076ad3ab5b9046b2bd88373d4be7b7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:20.248425 11757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d1869fab6f64b8ba73c1e4786ac9b6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:20.253444 12124 tablet_bootstrap.cc:492] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138: Bootstrap starting.
I20260621 02:44:20.256004 12125 tablet_bootstrap.cc:492] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc: Bootstrap starting.
I20260621 02:44:20.256878 12124 tablet_bootstrap.cc:654] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:20.258057 12124 log.cc:824] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:20.259080 12125 tablet_bootstrap.cc:654] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:20.260136 12125 log.cc:824] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:20.261956 12124 tablet_bootstrap.cc:492] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138: No bootstrap required, opened a new log
I20260621 02:44:20.262344 12124 ts_tablet_manager.cc:1397] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260621 02:44:20.262646 12125 tablet_bootstrap.cc:492] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc: No bootstrap required, opened a new log
I20260621 02:44:20.262880 12125 ts_tablet_manager.cc:1397] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260621 02:44:20.269363 12127 tablet_bootstrap.cc:492] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b: Bootstrap starting.
I20260621 02:44:20.270387 12125 raft_consensus.cc:348] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } } peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } }
I20260621 02:44:20.271065 12125 raft_consensus.cc:374] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:20.271198 12125 raft_consensus.cc:729] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fce76e8380224ee5b83154d8c9ba82dc, State: Initialized, Role: FOLLOWER
I20260621 02:44:20.271944 12125 consensus_queue.cc:260] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } } peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } }
I20260621 02:44:20.273010 12105 heartbeater.cc:503] Master 127.9.166.62:37717 was elected leader, sending a full tablet report...
I20260621 02:44:20.273209 12127 tablet_bootstrap.cc:654] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:20.274353 12127 log.cc:824] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:20.274969 12125 ts_tablet_manager.cc:1428] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20260621 02:44:20.275507 12124 raft_consensus.cc:348] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } } peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } }
I20260621 02:44:20.275898 12124 raft_consensus.cc:374] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:20.276012 12124 raft_consensus.cc:729] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc0f56a333b442d8f50f6a5eb21d138, State: Initialized, Role: FOLLOWER
I20260621 02:44:20.276558 12124 consensus_queue.cc:260] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } } peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } }
I20260621 02:44:20.277534 11966 heartbeater.cc:503] Master 127.9.166.62:37717 was elected leader, sending a full tablet report...
I20260621 02:44:20.278003 12124 ts_tablet_manager.cc:1428] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260621 02:44:20.278226 12127 tablet_bootstrap.cc:492] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b: No bootstrap required, opened a new log
I20260621 02:44:20.278514 12127 ts_tablet_manager.cc:1397] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260621 02:44:20.286919 12127 raft_consensus.cc:348] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } } peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } }
I20260621 02:44:20.287237 12127 raft_consensus.cc:374] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:20.287349 12127 raft_consensus.cc:729] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ea7d2e312c54d69a527f948299ed79b, State: Initialized, Role: FOLLOWER
I20260621 02:44:20.287940 12127 consensus_queue.cc:260] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } } peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } }
I20260621 02:44:20.289397 12127 ts_tablet_manager.cc:1428] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b: Time spent starting tablet: real 0.011s user 0.004s sys 0.006s
I20260621 02:44:20.322142 12129 raft_consensus.cc:482] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:44:20.322419 12129 raft_consensus.cc:504] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } } peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } }
I20260621 02:44:20.324044 12129 leader_election.cc:288] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbc0f56a333b442d8f50f6a5eb21d138 (127.9.166.2:34747), 1ea7d2e312c54d69a527f948299ed79b (127.9.166.1:42573)
I20260621 02:44:20.329701 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.324497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:57237 (local address 127.9.166.2:34747)
0621 02:44:20.324719 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:20.324725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:20.324960 (+ 235us) server_negotiation.cc:407] Connection header received
0621 02:44:20.325173 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:20.325185 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:20.325254 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:20.325353 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:20.325753 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.326464 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.328053 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.328811 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.328957 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.329057 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:20.329325 (+ 268us) server_negotiation.cc:299] Negotiation successful
0621 02:44:20.329446 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:44:20.329725 11994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.324539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:34747 (local address 127.9.166.3:57237)
0621 02:44:20.324860 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:20.324874 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:20.325028 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:20.325374 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:20.325378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:20.325404 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:20.325609 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.325617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.326609 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.326615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.327899 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.327911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.328970 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.328975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.329158 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.329181 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 02:44:20.329292 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 02:44:20.329421 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208}
I20260621 02:44:20.330492 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.324569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:41929 (local address 127.9.166.1:42573)
0621 02:44:20.324759 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:20.324778 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:20.325481 (+ 703us) server_negotiation.cc:407] Connection header received
0621 02:44:20.325691 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:20.325695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:20.325763 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:20.325853 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:20.326381 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.327041 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.328695 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.329458 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.329608 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.329713 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:20.330074 (+ 361us) server_negotiation.cc:299] Negotiation successful
0621 02:44:20.330237 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:44:20.330989 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.324564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:42573 (local address 127.9.166.3:41929)
0621 02:44:20.325356 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:20.325373 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:20.325537 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:20.325876 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:20.325881 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:20.325903 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:20.326196 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.326204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.327166 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.327173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.328536 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.328547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.329669 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.329675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.329907 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.329937 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 02:44:20.330659 (+ 722us) client_negotiation.cc:240] Negotiation successful
0621 02:44:20.330772 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":472,"threads_started":1}
I20260621 02:44:20.330868 11916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d1869fab6f64b8ba73c1e4786ac9b6e" candidate_uuid: "fce76e8380224ee5b83154d8c9ba82dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" is_pre_election: true
I20260621 02:44:20.331430 11916 raft_consensus.cc:2454] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fce76e8380224ee5b83154d8c9ba82dc in term 0.
I20260621 02:44:20.332072 11777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d1869fab6f64b8ba73c1e4786ac9b6e" candidate_uuid: "fce76e8380224ee5b83154d8c9ba82dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ea7d2e312c54d69a527f948299ed79b" is_pre_election: true
I20260621 02:44:20.332226 11987 leader_election.cc:302] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fbc0f56a333b442d8f50f6a5eb21d138, fce76e8380224ee5b83154d8c9ba82dc; no voters:
I20260621 02:44:20.332631 12129 raft_consensus.cc:2781] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:44:20.332638 11777 raft_consensus.cc:2454] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fce76e8380224ee5b83154d8c9ba82dc in term 0.
I20260621 02:44:20.332788 12129 raft_consensus.cc:482] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:44:20.332894 12129 raft_consensus.cc:3037] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:20.335136 12129 raft_consensus.cc:504] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } } peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } }
I20260621 02:44:20.335745 12129 leader_election.cc:288] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [CANDIDATE]: Term 1 election: Requested vote from peers fbc0f56a333b442d8f50f6a5eb21d138 (127.9.166.2:34747), 1ea7d2e312c54d69a527f948299ed79b (127.9.166.1:42573)
I20260621 02:44:20.336280 11777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d1869fab6f64b8ba73c1e4786ac9b6e" candidate_uuid: "fce76e8380224ee5b83154d8c9ba82dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ea7d2e312c54d69a527f948299ed79b"
I20260621 02:44:20.336310 11916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d1869fab6f64b8ba73c1e4786ac9b6e" candidate_uuid: "fce76e8380224ee5b83154d8c9ba82dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc0f56a333b442d8f50f6a5eb21d138"
I20260621 02:44:20.336494 11777 raft_consensus.cc:3037] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:20.336498 11916 raft_consensus.cc:3037] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:20.339689 11916 raft_consensus.cc:2454] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fce76e8380224ee5b83154d8c9ba82dc in term 1.
I20260621 02:44:20.339689 11777 raft_consensus.cc:2454] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fce76e8380224ee5b83154d8c9ba82dc in term 1.
I20260621 02:44:20.340075 11988 leader_election.cc:302] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ea7d2e312c54d69a527f948299ed79b, fce76e8380224ee5b83154d8c9ba82dc; no voters:
I20260621 02:44:20.340366 12129 raft_consensus.cc:2781] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:20.340755 12129 raft_consensus.cc:686] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [term 1 LEADER]: Becoming Leader. State: Replica: fce76e8380224ee5b83154d8c9ba82dc, State: Running, Role: LEADER
I20260621 02:44:20.341219 12129 consensus_queue.cc:237] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } } peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } }
I20260621 02:44:20.345840 11620 catalog_manager.cc:5495] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc reported cstate change: term changed from 0 to 1, leader changed from <none> to fce76e8380224ee5b83154d8c9ba82dc (127.9.166.3). New cstate: current_term: 1 leader_uuid: "fce76e8380224ee5b83154d8c9ba82dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } health_report { overall_health: UNKNOWN } } }
W20260621 02:44:20.390640 11967 tablet.cc:2369] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:44:20.394130 11828 tablet.cc:2369] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:44:20.395730 12106 tablet.cc:2369] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 21 02:44:20 dist-test-slave-hvgn krb5kdc[11575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009858, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:20.411039 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.397699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40612 (local address 127.9.166.62:37717)
0621 02:44:20.397869 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:20.397873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:20.398421 (+ 548us) server_negotiation.cc:407] Connection header received
0621 02:44:20.398618 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:20.398623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:20.398683 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:20.398780 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:20.399957 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.400650 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.401880 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.402312 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.402444 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.405484 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:20.405489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:20.405492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:20.405518 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:20.408295 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:20.408985 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:20.408990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:20.408994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:20.409115 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:20.409556 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:20.409561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:20.409564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:20.409885 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:20.410046 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:20.410436 (+ 390us) server_negotiation.cc:299] Negotiation successful
0621 02:44:20.410742 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":56}
I20260621 02:44:20.411074 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.397761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37717 (local address 127.0.0.1:40612)
0621 02:44:20.398289 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:20.398304 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:20.398494 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:20.398804 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:20.398808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:20.399559 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:20.399809 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.399819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.400772 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.400780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.401742 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.401754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.402485 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.402490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.402643 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.403240 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:20.403256 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:20.405290 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:20.408412 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:20.408425 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:20.408432 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:20.408867 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:20.409217 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:20.409223 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:20.409226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:20.409417 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:20.410006 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:20.410018 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:20.410181 (+ 163us) client_negotiation.cc:769] Sending connection context
0621 02:44:20.410343 (+ 162us) client_negotiation.cc:240] Negotiation successful
0621 02:44:20.410614 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":281,"threads_started":1}
I20260621 02:44:20.415576 11620 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:40612:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 02:44:20.416698 11620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:44:20.423113 11757 tablet_service.cc:1467] Processing CreateTablet for tablet be1a1cc871994b758ae6635e126c680c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=88c546fc6f07484aa3fc6b7a3a7c5b9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:44:20.423115 11896 tablet_service.cc:1467] Processing CreateTablet for tablet be1a1cc871994b758ae6635e126c680c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=88c546fc6f07484aa3fc6b7a3a7c5b9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:44:20.423563 12035 tablet_service.cc:1467] Processing CreateTablet for tablet be1a1cc871994b758ae6635e126c680c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=88c546fc6f07484aa3fc6b7a3a7c5b9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:44:20.424111 11896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1a1cc871994b758ae6635e126c680c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:20.424111 11757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1a1cc871994b758ae6635e126c680c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:20.424392 12035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1a1cc871994b758ae6635e126c680c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:20.429327 12125 tablet_bootstrap.cc:492] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc: Bootstrap starting.
I20260621 02:44:20.430945 12125 tablet_bootstrap.cc:654] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:20.431133 12127 tablet_bootstrap.cc:492] T be1a1cc871994b758ae6635e126c680c P 1ea7d2e312c54d69a527f948299ed79b: Bootstrap starting.
I20260621 02:44:20.431141 12124 tablet_bootstrap.cc:492] T be1a1cc871994b758ae6635e126c680c P fbc0f56a333b442d8f50f6a5eb21d138: Bootstrap starting.
I20260621 02:44:20.432723 12124 tablet_bootstrap.cc:654] T be1a1cc871994b758ae6635e126c680c P fbc0f56a333b442d8f50f6a5eb21d138: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:20.432801 12127 tablet_bootstrap.cc:654] T be1a1cc871994b758ae6635e126c680c P 1ea7d2e312c54d69a527f948299ed79b: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:20.433254 12125 tablet_bootstrap.cc:492] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc: No bootstrap required, opened a new log
I20260621 02:44:20.433408 12125 ts_tablet_manager.cc:1397] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260621 02:44:20.434325 12125 raft_consensus.cc:348] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:20.434578 12125 raft_consensus.cc:374] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:20.434695 12125 raft_consensus.cc:729] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fce76e8380224ee5b83154d8c9ba82dc, State: Initialized, Role: FOLLOWER
I20260621 02:44:20.434828 12124 tablet_bootstrap.cc:492] T be1a1cc871994b758ae6635e126c680c P fbc0f56a333b442d8f50f6a5eb21d138: No bootstrap required, opened a new log
I20260621 02:44:20.434965 12124 ts_tablet_manager.cc:1397] T be1a1cc871994b758ae6635e126c680c P fbc0f56a333b442d8f50f6a5eb21d138: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260621 02:44:20.434975 12125 consensus_queue.cc:260] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:20.435462 12129 tablet_replica.cc:439] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } } }
I20260621 02:44:20.435647 12125 ts_tablet_manager.cc:1428] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:44:20.435760 12129 tablet_replica.cc:442] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:44:20.435685 12124 raft_consensus.cc:348] T be1a1cc871994b758ae6635e126c680c P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:20.435947 12124 raft_consensus.cc:374] T be1a1cc871994b758ae6635e126c680c P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:20.436048 12124 raft_consensus.cc:729] T be1a1cc871994b758ae6635e126c680c P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc0f56a333b442d8f50f6a5eb21d138, State: Initialized, Role: FOLLOWER
I20260621 02:44:20.436307 12124 consensus_queue.cc:260] T be1a1cc871994b758ae6635e126c680c P fbc0f56a333b442d8f50f6a5eb21d138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:20.436914 12124 ts_tablet_manager.cc:1428] T be1a1cc871994b758ae6635e126c680c P fbc0f56a333b442d8f50f6a5eb21d138: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:44:20.437057 12127 tablet_bootstrap.cc:492] T be1a1cc871994b758ae6635e126c680c P 1ea7d2e312c54d69a527f948299ed79b: No bootstrap required, opened a new log
I20260621 02:44:20.436798 12131 tablet_replica.cc:439] T be1a1cc871994b758ae6635e126c680c P fbc0f56a333b442d8f50f6a5eb21d138: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } } }
I20260621 02:44:20.437172 12131 tablet_replica.cc:442] T be1a1cc871994b758ae6635e126c680c P fbc0f56a333b442d8f50f6a5eb21d138: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:44:20.437260 12127 ts_tablet_manager.cc:1397] T be1a1cc871994b758ae6635e126c680c P 1ea7d2e312c54d69a527f948299ed79b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 02:44:20.438053 12127 raft_consensus.cc:348] T be1a1cc871994b758ae6635e126c680c P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:20.438354 12127 raft_consensus.cc:374] T be1a1cc871994b758ae6635e126c680c P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:20.438491 12127 raft_consensus.cc:729] T be1a1cc871994b758ae6635e126c680c P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ea7d2e312c54d69a527f948299ed79b, State: Initialized, Role: FOLLOWER
I20260621 02:44:20.438845 12127 consensus_queue.cc:260] T be1a1cc871994b758ae6635e126c680c P 1ea7d2e312c54d69a527f948299ed79b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:20.439477 12127 ts_tablet_manager.cc:1428] T be1a1cc871994b758ae6635e126c680c P 1ea7d2e312c54d69a527f948299ed79b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:44:20.439568 12132 tablet_replica.cc:439] T be1a1cc871994b758ae6635e126c680c P 1ea7d2e312c54d69a527f948299ed79b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } } }
I20260621 02:44:20.439992 12132 tablet_replica.cc:442] T be1a1cc871994b758ae6635e126c680c P 1ea7d2e312c54d69a527f948299ed79b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:44:20.549494 12129 raft_consensus.cc:482] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:44:20.549746 12129 raft_consensus.cc:504] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:20.550546 12129 leader_election.cc:288] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ea7d2e312c54d69a527f948299ed79b (127.9.166.1:42573), fbc0f56a333b442d8f50f6a5eb21d138 (127.9.166.2:34747)
I20260621 02:44:20.551107 11916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1a1cc871994b758ae6635e126c680c" candidate_uuid: "fce76e8380224ee5b83154d8c9ba82dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" is_pre_election: true
I20260621 02:44:20.551105 11777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1a1cc871994b758ae6635e126c680c" candidate_uuid: "fce76e8380224ee5b83154d8c9ba82dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ea7d2e312c54d69a527f948299ed79b" is_pre_election: true
I20260621 02:44:20.551362 11916 raft_consensus.cc:2454] T be1a1cc871994b758ae6635e126c680c P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fce76e8380224ee5b83154d8c9ba82dc in term 0.
I20260621 02:44:20.551362 11777 raft_consensus.cc:2454] T be1a1cc871994b758ae6635e126c680c P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fce76e8380224ee5b83154d8c9ba82dc in term 0.
I20260621 02:44:20.551870 11988 leader_election.cc:302] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ea7d2e312c54d69a527f948299ed79b, fce76e8380224ee5b83154d8c9ba82dc; no voters:
I20260621 02:44:20.552155 12129 raft_consensus.cc:2781] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:44:20.552237 12129 raft_consensus.cc:482] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:44:20.552353 12129 raft_consensus.cc:3037] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:20.554447 12129 raft_consensus.cc:504] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:20.555032 12129 leader_election.cc:288] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc [CANDIDATE]: Term 1 election: Requested vote from peers 1ea7d2e312c54d69a527f948299ed79b (127.9.166.1:42573), fbc0f56a333b442d8f50f6a5eb21d138 (127.9.166.2:34747)
I20260621 02:44:20.555431 11777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1a1cc871994b758ae6635e126c680c" candidate_uuid: "fce76e8380224ee5b83154d8c9ba82dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ea7d2e312c54d69a527f948299ed79b"
I20260621 02:44:20.555588 11916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1a1cc871994b758ae6635e126c680c" candidate_uuid: "fce76e8380224ee5b83154d8c9ba82dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc0f56a333b442d8f50f6a5eb21d138"
I20260621 02:44:20.555642 11777 raft_consensus.cc:3037] T be1a1cc871994b758ae6635e126c680c P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:20.555819 11916 raft_consensus.cc:3037] T be1a1cc871994b758ae6635e126c680c P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:20.557667 11777 raft_consensus.cc:2454] T be1a1cc871994b758ae6635e126c680c P 1ea7d2e312c54d69a527f948299ed79b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fce76e8380224ee5b83154d8c9ba82dc in term 1.
I20260621 02:44:20.558070 11988 leader_election.cc:302] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ea7d2e312c54d69a527f948299ed79b, fce76e8380224ee5b83154d8c9ba82dc; no voters:
I20260621 02:44:20.558092 11916 raft_consensus.cc:2454] T be1a1cc871994b758ae6635e126c680c P fbc0f56a333b442d8f50f6a5eb21d138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fce76e8380224ee5b83154d8c9ba82dc in term 1.
I20260621 02:44:20.558295 12129 raft_consensus.cc:2781] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:20.558512 12129 raft_consensus.cc:686] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc [term 1 LEADER]: Becoming Leader. State: Replica: fce76e8380224ee5b83154d8c9ba82dc, State: Running, Role: LEADER
I20260621 02:44:20.558764 12129 consensus_queue.cc:237] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:20.560369 12134 tablet_replica.cc:439] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc: TxnStatusTablet state changed. Reason: New leader fce76e8380224ee5b83154d8c9ba82dc. Latest consensus state: current_term: 1 leader_uuid: "fce76e8380224ee5b83154d8c9ba82dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } } }
I20260621 02:44:20.560725 12134 tablet_replica.cc:442] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:44:20.561409 12146 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 02:44:20.561633 12146 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 02:44:20.561645 11618 catalog_manager.cc:5495] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc reported cstate change: term changed from 0 to 1, leader changed from <none> to fce76e8380224ee5b83154d8c9ba82dc (127.9.166.3). New cstate: current_term: 1 leader_uuid: "fce76e8380224ee5b83154d8c9ba82dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } health_report { overall_health: HEALTHY } } }
I20260621 02:44:20.593506 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.588078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44046 (local address 127.9.166.3:41199)
0621 02:44:20.588264 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:20.588272 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:20.588446 (+ 174us) server_negotiation.cc:407] Connection header received
0621 02:44:20.588729 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:20.588733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:20.588804 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:20.588954 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:44:20.589315 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.590044 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.591324 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.591812 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.591945 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.592408 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:44:20.592912 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:44:20.593057 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:20.593139 (+ 82us) server_negotiation.cc:299] Negotiation successful
0621 02:44:20.593211 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 02:44:20.593530 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.588132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:41199 (local address 127.0.0.1:44046)
0621 02:44:20.588357 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:20.588373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:20.588537 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:20.588971 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:20.588976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:20.588989 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:44:20.589178 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.589185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.590172 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.590180 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.591167 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.591178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.591959 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.591964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.592137 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.592184 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:44:20.593021 (+ 837us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:44:20.593039 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 02:44:20.593118 (+ 79us) client_negotiation.cc:240] Negotiation successful
0621 02:44:20.593256 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260621 02:44:20.601951 11777 raft_consensus.cc:1261] T be1a1cc871994b758ae6635e126c680c P 1ea7d2e312c54d69a527f948299ed79b [term 1 FOLLOWER]: Refusing update from remote peer fce76e8380224ee5b83154d8c9ba82dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:44:20.601934 11916 raft_consensus.cc:1261] T be1a1cc871994b758ae6635e126c680c P fbc0f56a333b442d8f50f6a5eb21d138 [term 1 FOLLOWER]: Refusing update from remote peer fce76e8380224ee5b83154d8c9ba82dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:44:20.602876 12134 consensus_queue.cc:1035] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:20.603185 12129 consensus_queue.cc:1035] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:20.610620 12132 tablet_replica.cc:439] T be1a1cc871994b758ae6635e126c680c P 1ea7d2e312c54d69a527f948299ed79b: TxnStatusTablet state changed. Reason: New leader fce76e8380224ee5b83154d8c9ba82dc. Latest consensus state: current_term: 1 leader_uuid: "fce76e8380224ee5b83154d8c9ba82dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } } }
I20260621 02:44:20.610925 12132 tablet_replica.cc:442] T be1a1cc871994b758ae6635e126c680c P 1ea7d2e312c54d69a527f948299ed79b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:44:20.613112 12149 mvcc.cc:205] Tried to move back new op lower bound from 7299112389016072192 to 7299112388851052544. Current Snapshot: MvccSnapshot[applied={T|T < 7299112389016072192}]
I20260621 02:44:20.624218 12129 tablet_replica.cc:439] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fce76e8380224ee5b83154d8c9ba82dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } } }
I20260621 02:44:20.624636 12129 tablet_replica.cc:442] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:44:20.627606 12131 tablet_replica.cc:439] T be1a1cc871994b758ae6635e126c680c P fbc0f56a333b442d8f50f6a5eb21d138: TxnStatusTablet state changed. Reason: New leader fce76e8380224ee5b83154d8c9ba82dc. Latest consensus state: current_term: 1 leader_uuid: "fce76e8380224ee5b83154d8c9ba82dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } } }
I20260621 02:44:20.627923 12131 tablet_replica.cc:442] T be1a1cc871994b758ae6635e126c680c P fbc0f56a333b442d8f50f6a5eb21d138: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:44:20.634572 12132 tablet_replica.cc:439] T be1a1cc871994b758ae6635e126c680c P 1ea7d2e312c54d69a527f948299ed79b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fce76e8380224ee5b83154d8c9ba82dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } } }
I20260621 02:44:20.634797 12132 tablet_replica.cc:442] T be1a1cc871994b758ae6635e126c680c P 1ea7d2e312c54d69a527f948299ed79b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:44:20.638530 12134 tablet_replica.cc:439] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fce76e8380224ee5b83154d8c9ba82dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } } }
I20260621 02:44:20.638880 12134 tablet_replica.cc:442] T be1a1cc871994b758ae6635e126c680c P fce76e8380224ee5b83154d8c9ba82dc: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:44:20.645313 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.637526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44062 (local address 127.9.166.3:41199)
0621 02:44:20.637698 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:20.637704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:20.637845 (+ 141us) server_negotiation.cc:407] Connection header received
0621 02:44:20.638082 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:20.638088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:20.638181 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:20.638263 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:44:20.638835 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.640255 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.642817 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.643484 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.643604 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.644204 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:44:20.644325 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:44:20.644413 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:20.644944 (+ 531us) server_negotiation.cc:299] Negotiation successful
0621 02:44:20.645027 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:44:20.665213 12131 tablet_replica.cc:439] T be1a1cc871994b758ae6635e126c680c P fbc0f56a333b442d8f50f6a5eb21d138: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fce76e8380224ee5b83154d8c9ba82dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } } }
I20260621 02:44:20.665474 12131 tablet_replica.cc:442] T be1a1cc871994b758ae6635e126c680c P fbc0f56a333b442d8f50f6a5eb21d138: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:44:20.675478 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.668908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:45497 (local address 127.9.166.3:41199)
0621 02:44:20.669096 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:20.669103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:20.669713 (+ 610us) server_negotiation.cc:407] Connection header received
0621 02:44:20.669945 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:20.669951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:20.670029 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:20.670124 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:44:20.670725 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.671953 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.673492 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.674170 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.674297 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.674696 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:44:20.674856 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:44:20.674953 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:20.675065 (+ 112us) server_negotiation.cc:299] Negotiation successful
0621 02:44:20.675139 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 02:44:20.675516 12161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.668929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:41199 (local address 127.9.166.3:45497)
0621 02:44:20.669623 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:20.669637 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:20.669793 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:20.670142 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:20.670145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:20.670157 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:44:20.670449 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.670458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.672098 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.672102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.673355 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.673367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.674339 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.674343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.674469 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.674520 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:44:20.674955 (+ 435us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:44:20.674962 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 02:44:20.675020 (+ 58us) client_negotiation.cc:240] Negotiation successful
0621 02:44:20.675111 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":398,"threads_started":1}
I20260621 02:44:20.685034 11777 raft_consensus.cc:1261] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b [term 1 FOLLOWER]: Refusing update from remote peer fce76e8380224ee5b83154d8c9ba82dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:44:20.685091 11916 raft_consensus.cc:1261] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138 [term 1 FOLLOWER]: Refusing update from remote peer fce76e8380224ee5b83154d8c9ba82dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:44:20.685557 12129 consensus_queue.cc:1035] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:20.685904 12157 consensus_queue.cc:1035] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:20.743795 11618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:44:20.744089 11618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:44:20.747571 11618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658: Sending DeleteTablet for 3 replicas of tablet 2d1869fab6f64b8ba73c1e4786ac9b6e
I20260621 02:44:20.748586 11896 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d1869fab6f64b8ba73c1e4786ac9b6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:20 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:38932
I20260621 02:44:20.748745 11757 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d1869fab6f64b8ba73c1e4786ac9b6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:20 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:60626
I20260621 02:44:20.750154 12035 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d1869fab6f64b8ba73c1e4786ac9b6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:20 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:44034
I20260621 02:44:20.751091 11618 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:44:20.752292 11618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:44:20.761145 12166 tablet_replica.cc:330] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b: stopping tablet replica
I20260621 02:44:20.762338 12166 raft_consensus.cc:2229] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:44:20.762580 11756 tablet_service.cc:1467] Processing CreateTablet for tablet cfb2b63092a644c9b53b9d70c976c90e (DEFAULT_TABLE table=test-table [id=780a2b68970040269e5a4e3639b901db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:20.763584 11756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfb2b63092a644c9b53b9d70c976c90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:20.763878 12034 tablet_service.cc:1467] Processing CreateTablet for tablet cfb2b63092a644c9b53b9d70c976c90e (DEFAULT_TABLE table=test-table [id=780a2b68970040269e5a4e3639b901db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:20.764575 12034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfb2b63092a644c9b53b9d70c976c90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:20.764801 12165 tablet_replica.cc:330] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138: stopping tablet replica
I20260621 02:44:20.765233 12165 raft_consensus.cc:2229] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:44:20.765689 12165 raft_consensus.cc:2258] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:20.765870 11895 tablet_service.cc:1467] Processing CreateTablet for tablet cfb2b63092a644c9b53b9d70c976c90e (DEFAULT_TABLE table=test-table [id=780a2b68970040269e5a4e3639b901db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:20.766360 11895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfb2b63092a644c9b53b9d70c976c90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:20.767772 12165 ts_tablet_manager.cc:1905] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:20.772846 12125 tablet_bootstrap.cc:492] T cfb2b63092a644c9b53b9d70c976c90e P fce76e8380224ee5b83154d8c9ba82dc: Bootstrap starting.
I20260621 02:44:20.774456 12125 tablet_bootstrap.cc:654] T cfb2b63092a644c9b53b9d70c976c90e P fce76e8380224ee5b83154d8c9ba82dc: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:20.775811 12165 ts_tablet_manager.cc:1918] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:44:20.775934 12167 tablet_replica.cc:330] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc: stopping tablet replica
I20260621 02:44:20.776058 12165 log.cc:1196] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/2d1869fab6f64b8ba73c1e4786ac9b6e
I20260621 02:44:20.776695 12165 ts_tablet_manager.cc:1939] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fbc0f56a333b442d8f50f6a5eb21d138: Deleting consensus metadata
I20260621 02:44:20.777629 12167 raft_consensus.cc:2229] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:44:20.778455 12167 raft_consensus.cc:2258] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:20.778661 11605 catalog_manager.cc:4841] TS fbc0f56a333b442d8f50f6a5eb21d138 (127.9.166.2:34747): tablet 2d1869fab6f64b8ba73c1e4786ac9b6e (table test-table [id=076ad3ab5b9046b2bd88373d4be7b7ad]) successfully deleted
I20260621 02:44:20.780009 12124 tablet_bootstrap.cc:492] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138: Bootstrap starting.
I20260621 02:44:20.780495 12167 ts_tablet_manager.cc:1905] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:20.780684 12127 tablet_bootstrap.cc:492] T cfb2b63092a644c9b53b9d70c976c90e P 1ea7d2e312c54d69a527f948299ed79b: Bootstrap starting.
I20260621 02:44:20.781672 12124 tablet_bootstrap.cc:654] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:20.782341 12127 tablet_bootstrap.cc:654] T cfb2b63092a644c9b53b9d70c976c90e P 1ea7d2e312c54d69a527f948299ed79b: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:20.787880 12124 tablet_bootstrap.cc:492] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138: No bootstrap required, opened a new log
I20260621 02:44:20.788132 12124 ts_tablet_manager.cc:1397] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260621 02:44:20.788923 12124 raft_consensus.cc:348] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:20.789284 12124 raft_consensus.cc:374] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:20.789397 12124 raft_consensus.cc:729] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc0f56a333b442d8f50f6a5eb21d138, State: Initialized, Role: FOLLOWER
I20260621 02:44:20.789728 12124 consensus_queue.cc:260] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:20.790385 12166 raft_consensus.cc:2258] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:20.791129 12167 ts_tablet_manager.cc:1918] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:44:20.791168 12127 tablet_bootstrap.cc:492] T cfb2b63092a644c9b53b9d70c976c90e P 1ea7d2e312c54d69a527f948299ed79b: No bootstrap required, opened a new log
I20260621 02:44:20.791347 12167 log.cc:1196] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/2d1869fab6f64b8ba73c1e4786ac9b6e
I20260621 02:44:20.791458 12127 ts_tablet_manager.cc:1397] T cfb2b63092a644c9b53b9d70c976c90e P 1ea7d2e312c54d69a527f948299ed79b: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260621 02:44:20.792030 12167 ts_tablet_manager.cc:1939] T 2d1869fab6f64b8ba73c1e4786ac9b6e P fce76e8380224ee5b83154d8c9ba82dc: Deleting consensus metadata
I20260621 02:44:20.792213 12127 raft_consensus.cc:348] T cfb2b63092a644c9b53b9d70c976c90e P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:20.792503 12127 raft_consensus.cc:374] T cfb2b63092a644c9b53b9d70c976c90e P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:20.792618 12127 raft_consensus.cc:729] T cfb2b63092a644c9b53b9d70c976c90e P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ea7d2e312c54d69a527f948299ed79b, State: Initialized, Role: FOLLOWER
I20260621 02:44:20.793030 12124 ts_tablet_manager.cc:1428] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260621 02:44:20.792900 12127 consensus_queue.cc:260] T cfb2b63092a644c9b53b9d70c976c90e P 1ea7d2e312c54d69a527f948299ed79b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:20.793483 12127 ts_tablet_manager.cc:1428] T cfb2b63092a644c9b53b9d70c976c90e P 1ea7d2e312c54d69a527f948299ed79b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260621 02:44:20.794791 11606 catalog_manager.cc:4841] TS fce76e8380224ee5b83154d8c9ba82dc (127.9.166.3:41199): tablet 2d1869fab6f64b8ba73c1e4786ac9b6e (table test-table [id=076ad3ab5b9046b2bd88373d4be7b7ad]) successfully deleted
I20260621 02:44:20.795264 12125 tablet_bootstrap.cc:492] T cfb2b63092a644c9b53b9d70c976c90e P fce76e8380224ee5b83154d8c9ba82dc: No bootstrap required, opened a new log
I20260621 02:44:20.795508 12125 ts_tablet_manager.cc:1397] T cfb2b63092a644c9b53b9d70c976c90e P fce76e8380224ee5b83154d8c9ba82dc: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.005s
I20260621 02:44:20.796342 12125 raft_consensus.cc:348] T cfb2b63092a644c9b53b9d70c976c90e P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:20.796633 12125 raft_consensus.cc:374] T cfb2b63092a644c9b53b9d70c976c90e P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:20.796675 12166 ts_tablet_manager.cc:1905] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:20.796751 12125 raft_consensus.cc:729] T cfb2b63092a644c9b53b9d70c976c90e P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fce76e8380224ee5b83154d8c9ba82dc, State: Initialized, Role: FOLLOWER
I20260621 02:44:20.797066 12125 consensus_queue.cc:260] T cfb2b63092a644c9b53b9d70c976c90e P fce76e8380224ee5b83154d8c9ba82dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:20.798211 12125 ts_tablet_manager.cc:1428] T cfb2b63092a644c9b53b9d70c976c90e P fce76e8380224ee5b83154d8c9ba82dc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260621 02:44:20.803632 12166 ts_tablet_manager.cc:1918] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:44:20.803771 12166 log.cc:1196] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/2d1869fab6f64b8ba73c1e4786ac9b6e
I20260621 02:44:20.804199 12166 ts_tablet_manager.cc:1939] T 2d1869fab6f64b8ba73c1e4786ac9b6e P 1ea7d2e312c54d69a527f948299ed79b: Deleting consensus metadata
I20260621 02:44:20.805526 11606 catalog_manager.cc:4841] TS 1ea7d2e312c54d69a527f948299ed79b (127.9.166.1:42573): tablet 2d1869fab6f64b8ba73c1e4786ac9b6e (table test-table [id=076ad3ab5b9046b2bd88373d4be7b7ad]) successfully deleted
I20260621 02:44:20.837015 12131 raft_consensus.cc:482] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:44:20.837248 12131 raft_consensus.cc:504] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:20.838686 12131 leader_election.cc:288] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ea7d2e312c54d69a527f948299ed79b (127.9.166.1:42573), fce76e8380224ee5b83154d8c9ba82dc (127.9.166.3:41199)
I20260621 02:44:20.846237 12172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.839084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:45233 (local address 127.9.166.1:42573)
0621 02:44:20.840211 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:20.840220 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:20.840241 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:44:20.840328 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:20.840334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:20.840408 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:20.840533 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:20.841070 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.842139 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.844094 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.845244 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.845372 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.845473 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:20.845757 (+ 284us) server_negotiation.cc:299] Negotiation successful
0621 02:44:20.845885 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":998,"thread_start_us":872,"threads_started":1}
I20260621 02:44:20.846267 12171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.838952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:42573 (local address 127.9.166.2:45233)
0621 02:44:20.839526 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:20.839552 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:20.839733 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:20.840575 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:20.840598 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:20.840640 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:20.840875 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.840886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.842262 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.842268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.843905 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.843923 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.845398 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.845404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.845588 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.845622 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 02:44:20.845724 (+ 102us) client_negotiation.cc:240] Negotiation successful
0621 02:44:20.845865 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"spinlock_wait_cycles":2944,"thread_start_us":248,"threads_started":1}
I20260621 02:44:20.847244 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.839974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:48521 (local address 127.9.166.3:41199)
0621 02:44:20.840182 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:20.840189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:20.840548 (+ 359us) server_negotiation.cc:407] Connection header received
0621 02:44:20.841177 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:20.841182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:20.841258 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:20.841350 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:20.841943 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.843052 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.844970 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.846102 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.846245 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.846355 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:20.846870 (+ 515us) server_negotiation.cc:299] Negotiation successful
0621 02:44:20.847005 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:44:20.847311 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.839602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:41199 (local address 127.9.166.2:48521)
0621 02:44:20.840462 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:20.840481 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:20.840671 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:20.841385 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:20.841390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:20.841414 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:20.841779 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.841789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.843194 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.843199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.844823 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.844834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.846500 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.846508 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.846699 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.846732 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 02:44:20.846828 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 02:44:20.846956 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"spinlock_wait_cycles":11392,"thread_start_us":590,"threads_started":1}
I20260621 02:44:20.847157 11776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb2b63092a644c9b53b9d70c976c90e" candidate_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ea7d2e312c54d69a527f948299ed79b" is_pre_election: true
I20260621 02:44:20.847503 11776 raft_consensus.cc:2454] T cfb2b63092a644c9b53b9d70c976c90e P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbc0f56a333b442d8f50f6a5eb21d138 in term 0.
I20260621 02:44:20.847996 11849 leader_election.cc:302] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ea7d2e312c54d69a527f948299ed79b, fbc0f56a333b442d8f50f6a5eb21d138; no voters:
I20260621 02:44:20.848191 12055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb2b63092a644c9b53b9d70c976c90e" candidate_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fce76e8380224ee5b83154d8c9ba82dc" is_pre_election: true
I20260621 02:44:20.848367 12131 raft_consensus.cc:2781] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:44:20.848492 12131 raft_consensus.cc:482] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:44:20.848618 12131 raft_consensus.cc:3037] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:20.848548 12055 raft_consensus.cc:2454] T cfb2b63092a644c9b53b9d70c976c90e P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbc0f56a333b442d8f50f6a5eb21d138 in term 0.
I20260621 02:44:20.851034 12131 raft_consensus.cc:504] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:20.851569 12131 leader_election.cc:288] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [CANDIDATE]: Term 1 election: Requested vote from peers 1ea7d2e312c54d69a527f948299ed79b (127.9.166.1:42573), fce76e8380224ee5b83154d8c9ba82dc (127.9.166.3:41199)
I20260621 02:44:20.851974 11776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb2b63092a644c9b53b9d70c976c90e" candidate_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ea7d2e312c54d69a527f948299ed79b"
I20260621 02:44:20.852175 11776 raft_consensus.cc:3037] T cfb2b63092a644c9b53b9d70c976c90e P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:20.852136 12055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb2b63092a644c9b53b9d70c976c90e" candidate_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fce76e8380224ee5b83154d8c9ba82dc"
I20260621 02:44:20.852334 12055 raft_consensus.cc:3037] T cfb2b63092a644c9b53b9d70c976c90e P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:20.854223 11776 raft_consensus.cc:2454] T cfb2b63092a644c9b53b9d70c976c90e P 1ea7d2e312c54d69a527f948299ed79b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbc0f56a333b442d8f50f6a5eb21d138 in term 1.
I20260621 02:44:20.854574 11849 leader_election.cc:302] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ea7d2e312c54d69a527f948299ed79b, fbc0f56a333b442d8f50f6a5eb21d138; no voters:
I20260621 02:44:20.854851 12131 raft_consensus.cc:2781] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:20.855378 12131 raft_consensus.cc:686] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [term 1 LEADER]: Becoming Leader. State: Replica: fbc0f56a333b442d8f50f6a5eb21d138, State: Running, Role: LEADER
I20260621 02:44:20.855577 12055 raft_consensus.cc:2454] T cfb2b63092a644c9b53b9d70c976c90e P fce76e8380224ee5b83154d8c9ba82dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbc0f56a333b442d8f50f6a5eb21d138 in term 1.
I20260621 02:44:20.856150 12131 consensus_queue.cc:237] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:20.859773 11618 catalog_manager.cc:5495] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbc0f56a333b442d8f50f6a5eb21d138 (127.9.166.2). New cstate: current_term: 1 leader_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } health_report { overall_health: UNKNOWN } } }
I20260621 02:44:20.889864 12176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.881871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38948 (local address 127.9.166.2:34747)
0621 02:44:20.883987 (+ 2116us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:20.883993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:20.884016 (+ 23us) server_negotiation.cc:407] Connection header received
0621 02:44:20.884113 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:20.884119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:20.884216 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:20.884392 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:44:20.884723 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.885932 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.887164 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.887838 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.887981 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.888416 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:44:20.889202 (+ 786us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:44:20.889349 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:20.889415 (+ 66us) server_negotiation.cc:299] Negotiation successful
0621 02:44:20.889499 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1947,"thread_start_us":1698,"threads_started":1}
I20260621 02:44:20.906287 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.900281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:41199 (local address 127.9.166.2:34059)
0621 02:44:20.900968 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:20.900985 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:20.901094 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:20.901428 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:20.901432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:20.901441 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:44:20.901695 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.901702 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.903120 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.903123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.904244 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.904255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.905237 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.905241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.905363 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.905410 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:44:20.905838 (+ 428us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:44:20.905845 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 02:44:20.905903 (+ 58us) client_negotiation.cc:240] Negotiation successful
0621 02:44:20.906000 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":376,"threads_started":1}
I20260621 02:44:20.906361 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.900298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:34059 (local address 127.9.166.3:41199)
0621 02:44:20.900509 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:20.900515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:20.901076 (+ 561us) server_negotiation.cc:407] Connection header received
0621 02:44:20.901218 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:20.901224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:20.901307 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:20.901413 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:44:20.901847 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.902994 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.904398 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.905075 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.905216 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.905595 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:44:20.905733 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:44:20.905837 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:20.905975 (+ 138us) server_negotiation.cc:299] Negotiation successful
0621 02:44:20.906059 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:44:20.915330 11776 raft_consensus.cc:1261] T cfb2b63092a644c9b53b9d70c976c90e P 1ea7d2e312c54d69a527f948299ed79b [term 1 FOLLOWER]: Refusing update from remote peer fbc0f56a333b442d8f50f6a5eb21d138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:44:20.915659 12055 raft_consensus.cc:1261] T cfb2b63092a644c9b53b9d70c976c90e P fce76e8380224ee5b83154d8c9ba82dc [term 1 FOLLOWER]: Refusing update from remote peer fbc0f56a333b442d8f50f6a5eb21d138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:44:20.916020 12174 consensus_queue.cc:1035] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:20.916700 12174 consensus_queue.cc:1035] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:20.928066 12151 mvcc.cc:205] Tried to move back new op lower bound from 7299112390302420992 to 7299112390069825536. Current Snapshot: MvccSnapshot[applied={T|T < 7299112390302420992}]
I20260621 02:44:20.951076 12176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.945351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:52231 (local address 127.9.166.2:34747)
0621 02:44:20.945697 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:20.945703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:20.945721 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:44:20.945784 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:20.945790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:20.945869 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:20.945992 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:44:20.946306 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.947284 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.949292 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.949859 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.949993 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.950454 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:44:20.950549 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:44:20.950615 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:20.950788 (+ 173us) server_negotiation.cc:299] Negotiation successful
0621 02:44:20.950844 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260621 02:44:20.951251 12161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:20.945003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:34747 (local address 127.9.166.3:52231)
0621 02:44:20.945201 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:20.945225 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:20.945339 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:20.946001 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:20.946005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:20.946014 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:44:20.946174 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.946181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.947470 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.947476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.949149 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:20.949166 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:20.950031 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:20.950037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:20.950259 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:20.950312 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:44:20.950654 (+ 342us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:44:20.950671 (+ 17us) client_negotiation.cc:769] Sending connection context
0621 02:44:20.950765 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 02:44:20.950906 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
W20260621 02:44:20.952399 12151 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260621 02:44:22.548491 11618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:44:22.548758 11618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:44:22.553069 11618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658: Sending DeleteTablet for 3 replicas of tablet cfb2b63092a644c9b53b9d70c976c90e
I20260621 02:44:22.553855 11756 tablet_service.cc:1514] Processing DeleteTablet for tablet cfb2b63092a644c9b53b9d70c976c90e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:22 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:60626
I20260621 02:44:22.553879 11896 tablet_service.cc:1514] Processing DeleteTablet for tablet cfb2b63092a644c9b53b9d70c976c90e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:22 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:38932
I20260621 02:44:22.554023 12035 tablet_service.cc:1514] Processing DeleteTablet for tablet cfb2b63092a644c9b53b9d70c976c90e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:22 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:44034
I20260621 02:44:22.572950 12207 tablet_replica.cc:330] T cfb2b63092a644c9b53b9d70c976c90e P 1ea7d2e312c54d69a527f948299ed79b: stopping tablet replica
I20260621 02:44:22.573153 12209 tablet_replica.cc:330] T cfb2b63092a644c9b53b9d70c976c90e P fce76e8380224ee5b83154d8c9ba82dc: stopping tablet replica
I20260621 02:44:22.573407 12208 tablet_replica.cc:330] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138: stopping tablet replica
I20260621 02:44:22.573421 12209 raft_consensus.cc:2229] T cfb2b63092a644c9b53b9d70c976c90e P fce76e8380224ee5b83154d8c9ba82dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:44:22.573626 12208 raft_consensus.cc:2229] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:44:22.573665 12209 raft_consensus.cc:2258] T cfb2b63092a644c9b53b9d70c976c90e P fce76e8380224ee5b83154d8c9ba82dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:22.574018 12207 raft_consensus.cc:2229] T cfb2b63092a644c9b53b9d70c976c90e P 1ea7d2e312c54d69a527f948299ed79b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:44:22.574235 12208 raft_consensus.cc:2258] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:22.574868 12209 ts_tablet_manager.cc:1905] T cfb2b63092a644c9b53b9d70c976c90e P fce76e8380224ee5b83154d8c9ba82dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:22.575134 12208 ts_tablet_manager.cc:1905] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:22.575641 12207 raft_consensus.cc:2258] T cfb2b63092a644c9b53b9d70c976c90e P 1ea7d2e312c54d69a527f948299ed79b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:22.576702 12207 ts_tablet_manager.cc:1905] T cfb2b63092a644c9b53b9d70c976c90e P 1ea7d2e312c54d69a527f948299ed79b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:22.582088 12208 ts_tablet_manager.cc:1918] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 02:44:22.582291 12208 log.cc:1196] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/cfb2b63092a644c9b53b9d70c976c90e
I20260621 02:44:22.582799 12208 ts_tablet_manager.cc:1939] T cfb2b63092a644c9b53b9d70c976c90e P fbc0f56a333b442d8f50f6a5eb21d138: Deleting consensus metadata
I20260621 02:44:22.583508 12207 ts_tablet_manager.cc:1918] T cfb2b63092a644c9b53b9d70c976c90e P 1ea7d2e312c54d69a527f948299ed79b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 02:44:22.583724 12207 log.cc:1196] T cfb2b63092a644c9b53b9d70c976c90e P 1ea7d2e312c54d69a527f948299ed79b: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/cfb2b63092a644c9b53b9d70c976c90e
I20260621 02:44:22.584192 12207 ts_tablet_manager.cc:1939] T cfb2b63092a644c9b53b9d70c976c90e P 1ea7d2e312c54d69a527f948299ed79b: Deleting consensus metadata
I20260621 02:44:22.584142 11605 catalog_manager.cc:4841] TS fbc0f56a333b442d8f50f6a5eb21d138 (127.9.166.2:34747): tablet cfb2b63092a644c9b53b9d70c976c90e (table test-table [id=780a2b68970040269e5a4e3639b901db]) successfully deleted
I20260621 02:44:22.585162 11606 catalog_manager.cc:4841] TS 1ea7d2e312c54d69a527f948299ed79b (127.9.166.1:42573): tablet cfb2b63092a644c9b53b9d70c976c90e (table test-table [id=780a2b68970040269e5a4e3639b901db]) successfully deleted
I20260621 02:44:22.585336 12209 ts_tablet_manager.cc:1918] T cfb2b63092a644c9b53b9d70c976c90e P fce76e8380224ee5b83154d8c9ba82dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 02:44:22.585476 12209 log.cc:1196] T cfb2b63092a644c9b53b9d70c976c90e P fce76e8380224ee5b83154d8c9ba82dc: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/cfb2b63092a644c9b53b9d70c976c90e
I20260621 02:44:22.585909 12209 ts_tablet_manager.cc:1939] T cfb2b63092a644c9b53b9d70c976c90e P fce76e8380224ee5b83154d8c9ba82dc: Deleting consensus metadata
I20260621 02:44:22.586992 11606 catalog_manager.cc:4841] TS fce76e8380224ee5b83154d8c9ba82dc (127.9.166.3:41199): tablet cfb2b63092a644c9b53b9d70c976c90e (table test-table [id=780a2b68970040269e5a4e3639b901db]) successfully deleted
Jun 21 02:44:22 dist-test-slave-hvgn krb5kdc[11575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009862, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 02:44:22 dist-test-slave-hvgn krb5kdc[11575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009862, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:22.616632 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:22.602820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51146 (local address 127.9.166.62:37717)
0621 02:44:22.603442 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:22.603450 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:22.603475 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:44:22.603653 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:22.603657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:22.603719 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:22.603811 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:22.605059 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:22.605898 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:22.607094 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:22.607545 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:22.607638 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:22.611499 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:22.611504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:22.611508 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:22.611547 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:22.614234 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:22.614883 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:22.614893 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:22.614895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:22.615014 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:22.615322 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:22.615325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:22.615327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:22.615668 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:22.615848 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:22.616099 (+ 251us) server_negotiation.cc:299] Negotiation successful
0621 02:44:22.616256 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":288,"threads_started":1}
I20260621 02:44:22.620139 11618 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:44:22.620800 11618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:44:22.627172 11756 tablet_service.cc:1467] Processing CreateTablet for tablet 7b17b1c385544eb0b7909175de06b7c6 (DEFAULT_TABLE table=test-table [id=319ba331493c45a9b6d9584c82a1633e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:22.627529 12035 tablet_service.cc:1467] Processing CreateTablet for tablet 7b17b1c385544eb0b7909175de06b7c6 (DEFAULT_TABLE table=test-table [id=319ba331493c45a9b6d9584c82a1633e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:22.627656 11756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b17b1c385544eb0b7909175de06b7c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:22.627961 12035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b17b1c385544eb0b7909175de06b7c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:22.628168 11896 tablet_service.cc:1467] Processing CreateTablet for tablet 7b17b1c385544eb0b7909175de06b7c6 (DEFAULT_TABLE table=test-table [id=319ba331493c45a9b6d9584c82a1633e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:22.628557 11896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b17b1c385544eb0b7909175de06b7c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:22.639679 12219 tablet_bootstrap.cc:492] T 7b17b1c385544eb0b7909175de06b7c6 P 1ea7d2e312c54d69a527f948299ed79b: Bootstrap starting.
I20260621 02:44:22.640306 12220 tablet_bootstrap.cc:492] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138: Bootstrap starting.
I20260621 02:44:22.641920 12219 tablet_bootstrap.cc:654] T 7b17b1c385544eb0b7909175de06b7c6 P 1ea7d2e312c54d69a527f948299ed79b: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:22.644716 12220 tablet_bootstrap.cc:654] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:22.647877 12218 tablet_bootstrap.cc:492] T 7b17b1c385544eb0b7909175de06b7c6 P fce76e8380224ee5b83154d8c9ba82dc: Bootstrap starting.
I20260621 02:44:22.649132 12218 tablet_bootstrap.cc:654] T 7b17b1c385544eb0b7909175de06b7c6 P fce76e8380224ee5b83154d8c9ba82dc: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:22.653345 12219 tablet_bootstrap.cc:492] T 7b17b1c385544eb0b7909175de06b7c6 P 1ea7d2e312c54d69a527f948299ed79b: No bootstrap required, opened a new log
I20260621 02:44:22.653539 12219 ts_tablet_manager.cc:1397] T 7b17b1c385544eb0b7909175de06b7c6 P 1ea7d2e312c54d69a527f948299ed79b: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.009s
I20260621 02:44:22.654544 12219 raft_consensus.cc:348] T 7b17b1c385544eb0b7909175de06b7c6 P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:22.654865 12219 raft_consensus.cc:374] T 7b17b1c385544eb0b7909175de06b7c6 P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:22.654968 12219 raft_consensus.cc:729] T 7b17b1c385544eb0b7909175de06b7c6 P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ea7d2e312c54d69a527f948299ed79b, State: Initialized, Role: FOLLOWER
I20260621 02:44:22.655297 12219 consensus_queue.cc:260] T 7b17b1c385544eb0b7909175de06b7c6 P 1ea7d2e312c54d69a527f948299ed79b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:22.656198 12218 tablet_bootstrap.cc:492] T 7b17b1c385544eb0b7909175de06b7c6 P fce76e8380224ee5b83154d8c9ba82dc: No bootstrap required, opened a new log
I20260621 02:44:22.656374 12218 ts_tablet_manager.cc:1397] T 7b17b1c385544eb0b7909175de06b7c6 P fce76e8380224ee5b83154d8c9ba82dc: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260621 02:44:22.657150 12218 raft_consensus.cc:348] T 7b17b1c385544eb0b7909175de06b7c6 P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:22.657450 12218 raft_consensus.cc:374] T 7b17b1c385544eb0b7909175de06b7c6 P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:22.657572 12218 raft_consensus.cc:729] T 7b17b1c385544eb0b7909175de06b7c6 P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fce76e8380224ee5b83154d8c9ba82dc, State: Initialized, Role: FOLLOWER
I20260621 02:44:22.657835 12218 consensus_queue.cc:260] T 7b17b1c385544eb0b7909175de06b7c6 P fce76e8380224ee5b83154d8c9ba82dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:22.660668 12218 ts_tablet_manager.cc:1428] T 7b17b1c385544eb0b7909175de06b7c6 P fce76e8380224ee5b83154d8c9ba82dc: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260621 02:44:22.662571 12219 ts_tablet_manager.cc:1428] T 7b17b1c385544eb0b7909175de06b7c6 P 1ea7d2e312c54d69a527f948299ed79b: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260621 02:44:22.664327 12220 tablet_bootstrap.cc:492] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138: No bootstrap required, opened a new log
I20260621 02:44:22.664505 12220 ts_tablet_manager.cc:1397] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.008s
I20260621 02:44:22.665431 12220 raft_consensus.cc:348] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:22.665696 12220 raft_consensus.cc:374] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:22.665802 12220 raft_consensus.cc:729] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc0f56a333b442d8f50f6a5eb21d138, State: Initialized, Role: FOLLOWER
I20260621 02:44:22.666064 12220 consensus_queue.cc:260] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:22.666548 12220 ts_tablet_manager.cc:1428] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260621 02:44:22.846025 12202 raft_consensus.cc:482] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:44:22.846262 12202 raft_consensus.cc:504] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:22.847054 12202 leader_election.cc:288] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ea7d2e312c54d69a527f948299ed79b (127.9.166.1:42573), fce76e8380224ee5b83154d8c9ba82dc (127.9.166.3:41199)
I20260621 02:44:22.847651 11777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b17b1c385544eb0b7909175de06b7c6" candidate_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ea7d2e312c54d69a527f948299ed79b" is_pre_election: true
I20260621 02:44:22.847747 12055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b17b1c385544eb0b7909175de06b7c6" candidate_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fce76e8380224ee5b83154d8c9ba82dc" is_pre_election: true
I20260621 02:44:22.847888 11777 raft_consensus.cc:2454] T 7b17b1c385544eb0b7909175de06b7c6 P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbc0f56a333b442d8f50f6a5eb21d138 in term 0.
I20260621 02:44:22.848066 12055 raft_consensus.cc:2454] T 7b17b1c385544eb0b7909175de06b7c6 P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbc0f56a333b442d8f50f6a5eb21d138 in term 0.
I20260621 02:44:22.848424 11849 leader_election.cc:302] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ea7d2e312c54d69a527f948299ed79b, fbc0f56a333b442d8f50f6a5eb21d138; no voters:
I20260621 02:44:22.848712 12202 raft_consensus.cc:2781] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:44:22.848836 12202 raft_consensus.cc:482] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:44:22.848929 12202 raft_consensus.cc:3037] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:22.850951 12202 raft_consensus.cc:504] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:22.851547 12202 leader_election.cc:288] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [CANDIDATE]: Term 1 election: Requested vote from peers 1ea7d2e312c54d69a527f948299ed79b (127.9.166.1:42573), fce76e8380224ee5b83154d8c9ba82dc (127.9.166.3:41199)
I20260621 02:44:22.851984 11777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b17b1c385544eb0b7909175de06b7c6" candidate_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ea7d2e312c54d69a527f948299ed79b"
I20260621 02:44:22.852087 12055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b17b1c385544eb0b7909175de06b7c6" candidate_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fce76e8380224ee5b83154d8c9ba82dc"
I20260621 02:44:22.852195 11777 raft_consensus.cc:3037] T 7b17b1c385544eb0b7909175de06b7c6 P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:22.852308 12055 raft_consensus.cc:3037] T 7b17b1c385544eb0b7909175de06b7c6 P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:22.854257 12055 raft_consensus.cc:2454] T 7b17b1c385544eb0b7909175de06b7c6 P fce76e8380224ee5b83154d8c9ba82dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbc0f56a333b442d8f50f6a5eb21d138 in term 1.
I20260621 02:44:22.854471 11777 raft_consensus.cc:2454] T 7b17b1c385544eb0b7909175de06b7c6 P 1ea7d2e312c54d69a527f948299ed79b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbc0f56a333b442d8f50f6a5eb21d138 in term 1.
I20260621 02:44:22.854689 11849 leader_election.cc:302] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fbc0f56a333b442d8f50f6a5eb21d138, fce76e8380224ee5b83154d8c9ba82dc; no voters:
I20260621 02:44:22.855019 12202 raft_consensus.cc:2781] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:22.855170 12202 raft_consensus.cc:686] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [term 1 LEADER]: Becoming Leader. State: Replica: fbc0f56a333b442d8f50f6a5eb21d138, State: Running, Role: LEADER
I20260621 02:44:22.855520 12202 consensus_queue.cc:237] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:22.858451 11620 catalog_manager.cc:5495] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbc0f56a333b442d8f50f6a5eb21d138 (127.9.166.2). New cstate: current_term: 1 leader_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } health_report { overall_health: UNKNOWN } } }
I20260621 02:44:22.886215 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:22.881538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56146 (local address 127.9.166.2:34747)
0621 02:44:22.882073 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:22.882078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:22.882097 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:44:22.882176 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:22.882180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:22.882262 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:22.882345 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:44:22.882660 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:22.883456 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:22.884675 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:22.885084 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:22.885177 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:22.885528 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:44:22.885624 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:44:22.885688 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:22.885806 (+ 118us) server_negotiation.cc:299] Negotiation successful
0621 02:44:22.885861 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":312,"threads_started":1}
I20260621 02:44:22.894351 12055 raft_consensus.cc:1261] T 7b17b1c385544eb0b7909175de06b7c6 P fce76e8380224ee5b83154d8c9ba82dc [term 1 FOLLOWER]: Refusing update from remote peer fbc0f56a333b442d8f50f6a5eb21d138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:44:22.894811 11777 raft_consensus.cc:1261] T 7b17b1c385544eb0b7909175de06b7c6 P 1ea7d2e312c54d69a527f948299ed79b [term 1 FOLLOWER]: Refusing update from remote peer fbc0f56a333b442d8f50f6a5eb21d138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:44:22.894927 12131 consensus_queue.cc:1035] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:22.895329 12202 consensus_queue.cc:1035] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:22.937095 11620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:44:22.937288 11620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:44:22.939997 11620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658: Sending DeleteTablet for 3 replicas of tablet 7b17b1c385544eb0b7909175de06b7c6
I20260621 02:44:22.940658 11756 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b17b1c385544eb0b7909175de06b7c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:22 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:60626
I20260621 02:44:22.940757 11896 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b17b1c385544eb0b7909175de06b7c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:22 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:38932
I20260621 02:44:22.940862 12035 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b17b1c385544eb0b7909175de06b7c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:22 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:44034
I20260621 02:44:22.941062 12208 tablet_replica.cc:330] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138: stopping tablet replica
I20260621 02:44:22.941059 12207 tablet_replica.cc:330] T 7b17b1c385544eb0b7909175de06b7c6 P 1ea7d2e312c54d69a527f948299ed79b: stopping tablet replica
I20260621 02:44:22.941131 12209 tablet_replica.cc:330] T 7b17b1c385544eb0b7909175de06b7c6 P fce76e8380224ee5b83154d8c9ba82dc: stopping tablet replica
I20260621 02:44:22.941253 12208 raft_consensus.cc:2229] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:44:22.941272 12207 raft_consensus.cc:2229] T 7b17b1c385544eb0b7909175de06b7c6 P 1ea7d2e312c54d69a527f948299ed79b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:44:22.941301 12209 raft_consensus.cc:2229] T 7b17b1c385544eb0b7909175de06b7c6 P fce76e8380224ee5b83154d8c9ba82dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:44:22.941476 12209 raft_consensus.cc:2258] T 7b17b1c385544eb0b7909175de06b7c6 P fce76e8380224ee5b83154d8c9ba82dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:22.941489 12207 raft_consensus.cc:2258] T 7b17b1c385544eb0b7909175de06b7c6 P 1ea7d2e312c54d69a527f948299ed79b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:22.941560 12208 raft_consensus.cc:2258] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:22.941941 11620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260621 02:44:22.942586 12209 ts_tablet_manager.cc:1905] T 7b17b1c385544eb0b7909175de06b7c6 P fce76e8380224ee5b83154d8c9ba82dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:22.942584 12208 ts_tablet_manager.cc:1905] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260621 02:44:22.942591 11620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:44:22.942775 12207 ts_tablet_manager.cc:1905] T 7b17b1c385544eb0b7909175de06b7c6 P 1ea7d2e312c54d69a527f948299ed79b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:22.948975 12208 ts_tablet_manager.cc:1918] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:44:22.948984 12207 ts_tablet_manager.cc:1918] T 7b17b1c385544eb0b7909175de06b7c6 P 1ea7d2e312c54d69a527f948299ed79b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:44:22.949120 12207 log.cc:1196] T 7b17b1c385544eb0b7909175de06b7c6 P 1ea7d2e312c54d69a527f948299ed79b: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/7b17b1c385544eb0b7909175de06b7c6
I20260621 02:44:22.949124 12208 log.cc:1196] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/7b17b1c385544eb0b7909175de06b7c6
I20260621 02:44:22.949589 12207 ts_tablet_manager.cc:1939] T 7b17b1c385544eb0b7909175de06b7c6 P 1ea7d2e312c54d69a527f948299ed79b: Deleting consensus metadata
I20260621 02:44:22.949942 11896 tablet_service.cc:1467] Processing CreateTablet for tablet b3b8628b4f644e16b9099181e4299afa (DEFAULT_TABLE table=test-table [id=b19dec62ba5643d189ae0d4327302782]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:22.950068 11756 tablet_service.cc:1467] Processing CreateTablet for tablet b3b8628b4f644e16b9099181e4299afa (DEFAULT_TABLE table=test-table [id=b19dec62ba5643d189ae0d4327302782]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:22.950038 12035 tablet_service.cc:1467] Processing CreateTablet for tablet b3b8628b4f644e16b9099181e4299afa (DEFAULT_TABLE table=test-table [id=b19dec62ba5643d189ae0d4327302782]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:22.950320 11896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b8628b4f644e16b9099181e4299afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:22.950434 11756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b8628b4f644e16b9099181e4299afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:22.950449 12035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b8628b4f644e16b9099181e4299afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:22.950632 11606 catalog_manager.cc:4841] TS 1ea7d2e312c54d69a527f948299ed79b (127.9.166.1:42573): tablet 7b17b1c385544eb0b7909175de06b7c6 (table test-table [id=319ba331493c45a9b6d9584c82a1633e]) successfully deleted
I20260621 02:44:22.952008 12208 ts_tablet_manager.cc:1939] T 7b17b1c385544eb0b7909175de06b7c6 P fbc0f56a333b442d8f50f6a5eb21d138: Deleting consensus metadata
I20260621 02:44:22.953022 11605 catalog_manager.cc:4841] TS fbc0f56a333b442d8f50f6a5eb21d138 (127.9.166.2:34747): tablet 7b17b1c385544eb0b7909175de06b7c6 (table test-table [id=319ba331493c45a9b6d9584c82a1633e]) successfully deleted
I20260621 02:44:22.955246 12209 ts_tablet_manager.cc:1918] T 7b17b1c385544eb0b7909175de06b7c6 P fce76e8380224ee5b83154d8c9ba82dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:44:22.955549 12209 log.cc:1196] T 7b17b1c385544eb0b7909175de06b7c6 P fce76e8380224ee5b83154d8c9ba82dc: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/7b17b1c385544eb0b7909175de06b7c6
I20260621 02:44:22.956003 12209 ts_tablet_manager.cc:1939] T 7b17b1c385544eb0b7909175de06b7c6 P fce76e8380224ee5b83154d8c9ba82dc: Deleting consensus metadata
I20260621 02:44:22.956856 12218 tablet_bootstrap.cc:492] T b3b8628b4f644e16b9099181e4299afa P fce76e8380224ee5b83154d8c9ba82dc: Bootstrap starting.
I20260621 02:44:22.956899 12220 tablet_bootstrap.cc:492] T b3b8628b4f644e16b9099181e4299afa P fbc0f56a333b442d8f50f6a5eb21d138: Bootstrap starting.
I20260621 02:44:22.957064 11606 catalog_manager.cc:4841] TS fce76e8380224ee5b83154d8c9ba82dc (127.9.166.3:41199): tablet 7b17b1c385544eb0b7909175de06b7c6 (table test-table [id=319ba331493c45a9b6d9584c82a1633e]) successfully deleted
I20260621 02:44:22.958379 12218 tablet_bootstrap.cc:654] T b3b8628b4f644e16b9099181e4299afa P fce76e8380224ee5b83154d8c9ba82dc: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:22.958436 12219 tablet_bootstrap.cc:492] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b: Bootstrap starting.
I20260621 02:44:22.958607 12220 tablet_bootstrap.cc:654] T b3b8628b4f644e16b9099181e4299afa P fbc0f56a333b442d8f50f6a5eb21d138: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:22.960084 12219 tablet_bootstrap.cc:654] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:22.960372 12218 tablet_bootstrap.cc:492] T b3b8628b4f644e16b9099181e4299afa P fce76e8380224ee5b83154d8c9ba82dc: No bootstrap required, opened a new log
I20260621 02:44:22.960520 12218 ts_tablet_manager.cc:1397] T b3b8628b4f644e16b9099181e4299afa P fce76e8380224ee5b83154d8c9ba82dc: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260621 02:44:22.960767 12220 tablet_bootstrap.cc:492] T b3b8628b4f644e16b9099181e4299afa P fbc0f56a333b442d8f50f6a5eb21d138: No bootstrap required, opened a new log
I20260621 02:44:22.960935 12220 ts_tablet_manager.cc:1397] T b3b8628b4f644e16b9099181e4299afa P fbc0f56a333b442d8f50f6a5eb21d138: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260621 02:44:22.961285 12218 raft_consensus.cc:348] T b3b8628b4f644e16b9099181e4299afa P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:22.961544 12218 raft_consensus.cc:374] T b3b8628b4f644e16b9099181e4299afa P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:22.961658 12218 raft_consensus.cc:729] T b3b8628b4f644e16b9099181e4299afa P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fce76e8380224ee5b83154d8c9ba82dc, State: Initialized, Role: FOLLOWER
I20260621 02:44:22.961570 12220 raft_consensus.cc:348] T b3b8628b4f644e16b9099181e4299afa P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:22.961808 12220 raft_consensus.cc:374] T b3b8628b4f644e16b9099181e4299afa P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:22.961961 12220 raft_consensus.cc:729] T b3b8628b4f644e16b9099181e4299afa P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc0f56a333b442d8f50f6a5eb21d138, State: Initialized, Role: FOLLOWER
I20260621 02:44:22.961916 12218 consensus_queue.cc:260] T b3b8628b4f644e16b9099181e4299afa P fce76e8380224ee5b83154d8c9ba82dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:22.962105 12219 tablet_bootstrap.cc:492] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b: No bootstrap required, opened a new log
I20260621 02:44:22.962248 12219 ts_tablet_manager.cc:1397] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260621 02:44:22.962234 12220 consensus_queue.cc:260] T b3b8628b4f644e16b9099181e4299afa P fbc0f56a333b442d8f50f6a5eb21d138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:22.962424 12218 ts_tablet_manager.cc:1428] T b3b8628b4f644e16b9099181e4299afa P fce76e8380224ee5b83154d8c9ba82dc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:44:22.962754 12220 ts_tablet_manager.cc:1428] T b3b8628b4f644e16b9099181e4299afa P fbc0f56a333b442d8f50f6a5eb21d138: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260621 02:44:22.962931 12219 raft_consensus.cc:348] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:22.963169 12219 raft_consensus.cc:374] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:22.963238 12219 raft_consensus.cc:729] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ea7d2e312c54d69a527f948299ed79b, State: Initialized, Role: FOLLOWER
I20260621 02:44:22.963552 12219 consensus_queue.cc:260] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:22.964087 12219 ts_tablet_manager.cc:1428] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:44:23.161222 12224 raft_consensus.cc:482] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:44:23.161455 12224 raft_consensus.cc:504] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:23.163187 12224 leader_election.cc:288] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbc0f56a333b442d8f50f6a5eb21d138 (127.9.166.2:34747), fce76e8380224ee5b83154d8c9ba82dc (127.9.166.3:41199)
I20260621 02:44:23.172178 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:23.163655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:49323 (local address 127.9.166.2:34747)
0621 02:44:23.163898 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:23.163905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:23.165027 (+ 1122us) server_negotiation.cc:407] Connection header received
0621 02:44:23.165309 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:23.165314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:23.165390 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:23.165488 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:23.166097 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:23.167115 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:23.169691 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:23.171014 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:23.171127 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:23.171233 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:23.171748 (+ 515us) server_negotiation.cc:299] Negotiation successful
0621 02:44:23.171880 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 02:44:23.172196 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:23.163703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:41937 (local address 127.9.166.3:41199)
0621 02:44:23.164464 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:23.164471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:23.164855 (+ 384us) server_negotiation.cc:407] Connection header received
0621 02:44:23.165087 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:23.165091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:23.165151 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:23.165265 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:23.165830 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:23.166997 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:23.169685 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:23.170994 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:23.171127 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:23.171295 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:23.171748 (+ 453us) server_negotiation.cc:299] Negotiation successful
0621 02:44:23.171880 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":457,"threads_started":1}
I20260621 02:44:23.172410 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:23.163509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:34747 (local address 127.9.166.1:49323)
0621 02:44:23.164944 (+ 1435us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:23.164958 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:23.165151 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:23.165532 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:23.165536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:23.165564 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:23.165951 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:23.165959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:23.167242 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:23.167246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:23.169527 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:23.169535 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:23.171236 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:23.171246 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:23.171537 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:23.171608 (+ 71us) client_negotiation.cc:769] Sending connection context
0621 02:44:23.171704 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 02:44:23.171928 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1309,"mutex_wait_us":544,"thread_start_us":684,"threads_started":1}
I20260621 02:44:23.172724 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:23.163509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:41199 (local address 127.9.166.1:41937)
0621 02:44:23.164714 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:23.164739 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:23.164903 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:23.165277 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:23.165281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:23.165311 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:23.165634 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:23.165650 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:23.167122 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:23.167125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:23.169520 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:23.169535 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:23.171285 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:23.171291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:23.171554 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:23.171609 (+ 55us) client_negotiation.cc:769] Sending connection context
0621 02:44:23.171704 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 02:44:23.171928 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"mutex_wait_us":177,"thread_start_us":854,"threads_started":1}
I20260621 02:44:23.173081 11916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b8628b4f644e16b9099181e4299afa" candidate_uuid: "1ea7d2e312c54d69a527f948299ed79b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" is_pre_election: true
I20260621 02:44:23.173312 11916 raft_consensus.cc:2454] T b3b8628b4f644e16b9099181e4299afa P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ea7d2e312c54d69a527f948299ed79b in term 0.
I20260621 02:44:23.173344 12055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b8628b4f644e16b9099181e4299afa" candidate_uuid: "1ea7d2e312c54d69a527f948299ed79b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fce76e8380224ee5b83154d8c9ba82dc" is_pre_election: true
I20260621 02:44:23.173615 12055 raft_consensus.cc:2454] T b3b8628b4f644e16b9099181e4299afa P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ea7d2e312c54d69a527f948299ed79b in term 0.
I20260621 02:44:23.173774 11708 leader_election.cc:302] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ea7d2e312c54d69a527f948299ed79b, fbc0f56a333b442d8f50f6a5eb21d138; no voters:
I20260621 02:44:23.174156 12224 raft_consensus.cc:2781] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:44:23.174273 12224 raft_consensus.cc:482] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:44:23.174367 12224 raft_consensus.cc:3037] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:23.176497 12224 raft_consensus.cc:504] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:23.177079 12224 leader_election.cc:288] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [CANDIDATE]: Term 1 election: Requested vote from peers fbc0f56a333b442d8f50f6a5eb21d138 (127.9.166.2:34747), fce76e8380224ee5b83154d8c9ba82dc (127.9.166.3:41199)
I20260621 02:44:23.177479 11916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b8628b4f644e16b9099181e4299afa" candidate_uuid: "1ea7d2e312c54d69a527f948299ed79b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc0f56a333b442d8f50f6a5eb21d138"
I20260621 02:44:23.177525 12055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b8628b4f644e16b9099181e4299afa" candidate_uuid: "1ea7d2e312c54d69a527f948299ed79b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fce76e8380224ee5b83154d8c9ba82dc"
I20260621 02:44:23.177699 12055 raft_consensus.cc:3037] T b3b8628b4f644e16b9099181e4299afa P fce76e8380224ee5b83154d8c9ba82dc [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:23.177698 11916 raft_consensus.cc:3037] T b3b8628b4f644e16b9099181e4299afa P fbc0f56a333b442d8f50f6a5eb21d138 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:23.180684 11916 raft_consensus.cc:2454] T b3b8628b4f644e16b9099181e4299afa P fbc0f56a333b442d8f50f6a5eb21d138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ea7d2e312c54d69a527f948299ed79b in term 1.
I20260621 02:44:23.180693 12055 raft_consensus.cc:2454] T b3b8628b4f644e16b9099181e4299afa P fce76e8380224ee5b83154d8c9ba82dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ea7d2e312c54d69a527f948299ed79b in term 1.
I20260621 02:44:23.181121 11709 leader_election.cc:302] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ea7d2e312c54d69a527f948299ed79b, fce76e8380224ee5b83154d8c9ba82dc; no voters:
I20260621 02:44:23.181435 12224 raft_consensus.cc:2781] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:23.181918 12224 raft_consensus.cc:686] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [term 1 LEADER]: Becoming Leader. State: Replica: 1ea7d2e312c54d69a527f948299ed79b, State: Running, Role: LEADER
I20260621 02:44:23.182278 12224 consensus_queue.cc:237] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } }
I20260621 02:44:23.185465 11619 catalog_manager.cc:5495] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ea7d2e312c54d69a527f948299ed79b (127.9.166.1). New cstate: current_term: 1 leader_uuid: "1ea7d2e312c54d69a527f948299ed79b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ea7d2e312c54d69a527f948299ed79b" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 } health_report { overall_health: UNKNOWN } } }
I20260621 02:44:23.205034 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:23.199879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37092 (local address 127.9.166.1:42573)
0621 02:44:23.200395 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:23.200399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:23.200416 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:44:23.200493 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:23.200497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:23.200565 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:23.200734 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:44:23.201024 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:23.201881 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:23.203092 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:23.203537 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:23.203625 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:23.203998 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:44:23.204546 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:44:23.204648 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:23.204730 (+ 82us) server_negotiation.cc:299] Negotiation successful
0621 02:44:23.204785 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":264,"threads_started":1}
I20260621 02:44:23.219501 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:23.214400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:38943 (local address 127.9.166.3:41199)
0621 02:44:23.214567 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:23.214572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:23.215133 (+ 561us) server_negotiation.cc:407] Connection header received
0621 02:44:23.215273 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:23.215277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:23.215334 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:23.215459 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:44:23.215929 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:23.216584 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:23.218022 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:23.218387 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:23.218473 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:23.218850 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:44:23.218950 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:44:23.219011 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:23.219172 (+ 161us) server_negotiation.cc:299] Negotiation successful
0621 02:44:23.219239 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:44:23.219630 12243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:23.214205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:41199 (local address 127.9.166.1:38943)
0621 02:44:23.215044 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:23.215061 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:23.215172 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:23.215508 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:23.215511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:23.215521 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:44:23.215792 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:23.215800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:23.216706 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:23.216711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:23.217902 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:23.217915 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:23.218539 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:23.218544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:23.218666 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:23.218715 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:44:23.219059 (+ 344us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:44:23.219069 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 02:44:23.219139 (+ 70us) client_negotiation.cc:240] Negotiation successful
0621 02:44:23.219236 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":475,"threads_started":1}
I20260621 02:44:23.227074 12055 raft_consensus.cc:1261] T b3b8628b4f644e16b9099181e4299afa P fce76e8380224ee5b83154d8c9ba82dc [term 1 FOLLOWER]: Refusing update from remote peer 1ea7d2e312c54d69a527f948299ed79b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:44:23.227164 11916 raft_consensus.cc:1261] T b3b8628b4f644e16b9099181e4299afa P fbc0f56a333b442d8f50f6a5eb21d138 [term 1 FOLLOWER]: Refusing update from remote peer 1ea7d2e312c54d69a527f948299ed79b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:44:23.227748 12238 consensus_queue.cc:1035] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fce76e8380224ee5b83154d8c9ba82dc" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:23.228055 12224 consensus_queue.cc:1035] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbc0f56a333b442d8f50f6a5eb21d138" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:23.251286 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:23.246027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:35049 (local address 127.9.166.1:42573)
0621 02:44:23.246194 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:23.246201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:23.246656 (+ 455us) server_negotiation.cc:407] Connection header received
0621 02:44:23.246795 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:23.246798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:23.246853 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:23.246921 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:44:23.247299 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:23.248186 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:23.249898 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:23.250299 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:23.250394 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:23.250745 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:44:23.250837 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:44:23.250905 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:23.251015 (+ 110us) server_negotiation.cc:299] Negotiation successful
0621 02:44:23.251076 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260621 02:44:23.251315 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:23.245977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:42573 (local address 127.9.166.3:35049)
0621 02:44:23.246549 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:23.246568 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:23.246699 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:23.246947 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:23.246950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:23.246959 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:44:23.247168 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:23.247175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:23.248320 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:23.248323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:23.249762 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:23.249777 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:23.250438 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:23.250442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:23.250563 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:23.250598 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:44:23.250919 (+ 321us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:44:23.250926 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 02:44:23.250982 (+ 56us) client_negotiation.cc:240] Negotiation successful
0621 02:44:23.251071 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":269,"threads_started":1}
W20260621 02:44:23.252578 12189 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260621 02:44:24.846316 11618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:44:24.846529 11618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:44:24.850404 11618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c038bcc03ac44463a728929ccc853658: Sending DeleteTablet for 3 replicas of tablet b3b8628b4f644e16b9099181e4299afa
I20260621 02:44:24.851117 11756 tablet_service.cc:1514] Processing DeleteTablet for tablet b3b8628b4f644e16b9099181e4299afa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:24 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:60626
I20260621 02:44:24.851297 11896 tablet_service.cc:1514] Processing DeleteTablet for tablet b3b8628b4f644e16b9099181e4299afa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:24 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:38932
I20260621 02:44:24.851322 12035 tablet_service.cc:1514] Processing DeleteTablet for tablet b3b8628b4f644e16b9099181e4299afa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:24 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:44034
I20260621 02:44:24.852092 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 11691
I20260621 02:44:24.864845 12275 tablet_replica.cc:330] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b: stopping tablet replica
I20260621 02:44:24.865267 12275 raft_consensus.cc:2229] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:44:24.866053 12275 raft_consensus.cc:2258] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:24.866609 12276 tablet_replica.cc:330] T b3b8628b4f644e16b9099181e4299afa P fbc0f56a333b442d8f50f6a5eb21d138: stopping tablet replica
I20260621 02:44:24.866832 12277 tablet_replica.cc:330] T b3b8628b4f644e16b9099181e4299afa P fce76e8380224ee5b83154d8c9ba82dc: stopping tablet replica
I20260621 02:44:24.866932 12276 raft_consensus.cc:2229] T b3b8628b4f644e16b9099181e4299afa P fbc0f56a333b442d8f50f6a5eb21d138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:44:24.867210 12275 ts_tablet_manager.cc:1905] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:24.867228 12276 raft_consensus.cc:2258] T b3b8628b4f644e16b9099181e4299afa P fbc0f56a333b442d8f50f6a5eb21d138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:24.867210 12277 raft_consensus.cc:2229] T b3b8628b4f644e16b9099181e4299afa P fce76e8380224ee5b83154d8c9ba82dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:44:24.868292 12277 raft_consensus.cc:2258] T b3b8628b4f644e16b9099181e4299afa P fce76e8380224ee5b83154d8c9ba82dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:24.868750 12276 ts_tablet_manager.cc:1905] T b3b8628b4f644e16b9099181e4299afa P fbc0f56a333b442d8f50f6a5eb21d138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:24.869360 12277 ts_tablet_manager.cc:1905] T b3b8628b4f644e16b9099181e4299afa P fce76e8380224ee5b83154d8c9ba82dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:24.874491 12275 ts_tablet_manager.cc:1918] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 02:44:24.874706 12275 log.cc:1196] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/b3b8628b4f644e16b9099181e4299afa
I20260621 02:44:24.875260 12275 ts_tablet_manager.cc:1939] T b3b8628b4f644e16b9099181e4299afa P 1ea7d2e312c54d69a527f948299ed79b: Deleting consensus metadata
I20260621 02:44:24.875648 12276 ts_tablet_manager.cc:1918] T b3b8628b4f644e16b9099181e4299afa P fbc0f56a333b442d8f50f6a5eb21d138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 02:44:24.875818 12276 log.cc:1196] T b3b8628b4f644e16b9099181e4299afa P fbc0f56a333b442d8f50f6a5eb21d138: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/b3b8628b4f644e16b9099181e4299afa
I20260621 02:44:24.875849 12277 ts_tablet_manager.cc:1918] T b3b8628b4f644e16b9099181e4299afa P fce76e8380224ee5b83154d8c9ba82dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260621 02:44:24.876036 12277 log.cc:1196] T b3b8628b4f644e16b9099181e4299afa P fce76e8380224ee5b83154d8c9ba82dc: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/b3b8628b4f644e16b9099181e4299afa
I20260621 02:44:24.876320 12276 ts_tablet_manager.cc:1939] T b3b8628b4f644e16b9099181e4299afa P fbc0f56a333b442d8f50f6a5eb21d138: Deleting consensus metadata
I20260621 02:44:24.876571 12277 ts_tablet_manager.cc:1939] T b3b8628b4f644e16b9099181e4299afa P fce76e8380224ee5b83154d8c9ba82dc: Deleting consensus metadata
I20260621 02:44:24.876713 11606 catalog_manager.cc:4841] TS 1ea7d2e312c54d69a527f948299ed79b (127.9.166.1:42573): tablet b3b8628b4f644e16b9099181e4299afa (table test-table [id=b19dec62ba5643d189ae0d4327302782]) successfully deleted
I20260621 02:44:24.877401 11605 catalog_manager.cc:4841] TS fbc0f56a333b442d8f50f6a5eb21d138 (127.9.166.2:34747): tablet b3b8628b4f644e16b9099181e4299afa (table test-table [id=b19dec62ba5643d189ae0d4327302782]) successfully deleted
I20260621 02:44:24.877698 11606 catalog_manager.cc:4841] TS fce76e8380224ee5b83154d8c9ba82dc (127.9.166.3:41199): tablet b3b8628b4f644e16b9099181e4299afa (table test-table [id=b19dec62ba5643d189ae0d4327302782]) successfully deleted
Jun 21 02:44:24 dist-test-slave-hvgn krb5kdc[11575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009862, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:44:24.884418 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:24.853787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37094 (local address 127.9.166.1:42573)
0621 02:44:24.867779 (+ 13992us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:24.867788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:24.868444 (+ 656us) server_negotiation.cc:407] Connection header received
0621 02:44:24.868508 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:24.868513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:24.868606 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:24.868742 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:24.870489 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:24.871657 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:24.873518 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:24.874156 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:24.874287 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:24.879176 (+ 4889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:24.879189 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:24.879205 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:24.879239 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:24.882218 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:24.882771 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:24.882778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:24.882781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:24.882909 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:24.883213 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:24.883216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:24.883218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:24.883615 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:24.883819 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:24.884025 (+ 206us) server_negotiation.cc:299] Negotiation successful
0621 02:44:24.884174 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13841,"thread_start_us":13703,"threads_started":1}
W20260621 02:44:24.885023 11822 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37094
W20260621 02:44:24.885584 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 11691: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:44:24.885800 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 11691
I20260621 02:44:24.907438 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 11831
Jun 21 02:44:24 dist-test-slave-hvgn krb5kdc[11575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009862, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.166.2@KRBTEST.COM
I20260621 02:44:24.922714 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:24.908101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56160 (local address 127.9.166.2:34747)
0621 02:44:24.908581 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:24.908585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:24.908602 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:44:24.908653 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:24.908657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:24.908712 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:24.908809 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:24.910111 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:24.910947 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:24.912216 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:24.912656 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:24.912755 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:24.917349 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:24.917362 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:24.917382 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:24.917416 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:24.920278 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:24.920852 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:24.920860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:24.920862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:24.921038 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:24.921430 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:24.921434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:24.921436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:24.921760 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:24.922024 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:24.922280 (+ 256us) server_negotiation.cc:299] Negotiation successful
0621 02:44:24.922448 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":269,"threads_started":1}
W20260621 02:44:24.923416 11961 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56160
W20260621 02:44:24.924046 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 11831: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:44:24.924196 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 11831
I20260621 02:44:24.945609 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 11970
Jun 21 02:44:24 dist-test-slave-hvgn krb5kdc[11575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009862, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.166.3@KRBTEST.COM
I20260621 02:44:24.959942 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:24.946300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36756 (local address 127.9.166.3:41199)
0621 02:44:24.946801 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:24.946805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:24.946820 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:44:24.946873 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:24.946877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:24.946938 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:24.947026 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:24.948305 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:24.949053 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:24.950202 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:24.950604 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:24.950693 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:24.954763 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:24.954776 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:24.954792 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:24.954825 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:24.957549 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:24.958122 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:24.958129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:24.958131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:24.958271 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:24.958584 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:24.958587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:24.958589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:24.958887 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:24.959097 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:24.959289 (+ 192us) server_negotiation.cc:299] Negotiation successful
0621 02:44:24.959570 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":278,"threads_started":1}
W20260621 02:44:24.960583 12100 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36756
W20260621 02:44:24.961196 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 11970: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:44:24.961333 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 11970
I20260621 02:44:24.986578 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 11588
I20260621 02:44:24.987775 11660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:25.103307 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 11588
2026-06-21T02:44:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8276 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:44:25 dist-test-slave-hvgn krb5kdc[12285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:44:25 dist-test-slave-hvgn krb5kdc[12285](info): set up 2 sockets
Jun 21 02:44:25 dist-test-slave-hvgn krb5kdc[12285](info): commencing operation
krb5kdc: starting...
W20260621 02:44:26.187561 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:44:26 dist-test-slave-hvgn krb5kdc[12285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:44:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:44:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:44:26.366533 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:43643
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:34775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:43643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:26.676118 12298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:26.676436 12298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:26.676601 12298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:26.683611 12298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:44:26.683748 12298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:26.683859 12298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:26.683962 12298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:44:26.684049 12298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:44:26.693356 12298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:34775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:43643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:43643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:26.694621 12298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:26.695932 12298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:26.707633 12303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:26.707680 12304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:26.707742 12306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:26.708336 12298 server_base.cc:1034] running on GCE node
I20260621 02:44:26.709112 12298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:26.710397 12298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:26.711614 12298 hybrid_clock.cc:648] HybridClock initialized: now 1782009866711583 us; error 44 us; skew 500 ppm
I20260621 02:44:26.711977 12298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:26 dist-test-slave-hvgn krb5kdc[12285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009866, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:26.714989 12298 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:44:26.716405 12298 webserver.cc:460] Webserver started at http://127.9.166.62:46125/ using document root <none> and password file <none>
I20260621 02:44:26.717069 12298 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:26.717206 12298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:26.717505 12298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:26.720378 12298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "9c735ea0de3547d88bf2a9f45cd4bfd8"
format_stamp: "Formatted at 2026-06-21 02:44:26 on dist-test-slave-hvgn"
server_key: "f4cef4551badffa49a77715a9995e6db"
server_key_iv: "2e9d8bf6754808f10708aa5d7f7be1af"
server_key_version: "encryptionkey@0"
I20260621 02:44:26.721128 12298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "9c735ea0de3547d88bf2a9f45cd4bfd8"
format_stamp: "Formatted at 2026-06-21 02:44:26 on dist-test-slave-hvgn"
server_key: "f4cef4551badffa49a77715a9995e6db"
server_key_iv: "2e9d8bf6754808f10708aa5d7f7be1af"
server_key_version: "encryptionkey@0"
I20260621 02:44:26.725994 12298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260621 02:44:26.729276 12313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:26.730558 12298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:44:26.730769 12298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "9c735ea0de3547d88bf2a9f45cd4bfd8"
format_stamp: "Formatted at 2026-06-21 02:44:26 on dist-test-slave-hvgn"
server_key: "f4cef4551badffa49a77715a9995e6db"
server_key_iv: "2e9d8bf6754808f10708aa5d7f7be1af"
server_key_version: "encryptionkey@0"
I20260621 02:44:26.730966 12298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:26.745481 12298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:26.749600 12298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:26.749873 12298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:26.768280 12298 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:43643
I20260621 02:44:26.768316 12364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:43643 every 8 connection(s)
I20260621 02:44:26.769663 12298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:44:26.771250 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 12298
I20260621 02:44:26.771520 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:44:26.772307 9880 external_mini_cluster.cc:1428] Setting key dee4de7f3187d58eb05d5b70b3bfccf1
I20260621 02:44:26.774482 12365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:44:26 dist-test-slave-hvgn krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:26.787735 12365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8: Bootstrap starting.
I20260621 02:44:26.790727 12365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:26.791796 12365 log.cc:824] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:26.794544 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:26.774300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38226 (local address 127.9.166.62:43643)
0621 02:44:26.775113 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:26.775125 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:26.775167 (+ 42us) server_negotiation.cc:407] Connection header received
0621 02:44:26.776788 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:26.776809 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:26.777185 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:26.777512 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:26.779228 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:26.780611 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:26.782347 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:26.783093 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:26.783216 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:26.787679 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:26.787686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:26.787700 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:26.787741 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:26.790674 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:26.791364 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:26.791372 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:26.791426 (+ 54us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:26.791608 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:26.792022 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:26.792026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:26.792028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:26.792682 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:26.793007 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:26.793284 (+ 277us) server_negotiation.cc:299] Negotiation successful
0621 02:44:26.793569 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":423,"threads_started":1}
I20260621 02:44:26.794759 12365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8: No bootstrap required, opened a new log
I20260621 02:44:26.801818 12365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c735ea0de3547d88bf2a9f45cd4bfd8" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 43643 } }
I20260621 02:44:26.802163 12365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:26.802266 12365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c735ea0de3547d88bf2a9f45cd4bfd8, State: Initialized, Role: FOLLOWER
I20260621 02:44:26.802750 12365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c735ea0de3547d88bf2a9f45cd4bfd8" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 43643 } }
I20260621 02:44:26.802987 12365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:44:26.803119 12365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:44:26.803284 12365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:26.805511 12365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c735ea0de3547d88bf2a9f45cd4bfd8" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 43643 } }
I20260621 02:44:26.805990 12365 leader_election.cc:302] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c735ea0de3547d88bf2a9f45cd4bfd8; no voters:
I20260621 02:44:26.806561 12365 leader_election.cc:288] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:44:26.806696 12370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:26.807449 12370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8 [term 1 LEADER]: Becoming Leader. State: Replica: 9c735ea0de3547d88bf2a9f45cd4bfd8, State: Running, Role: LEADER
I20260621 02:44:26.807965 12370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c735ea0de3547d88bf2a9f45cd4bfd8" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 43643 } }
I20260621 02:44:26.808537 12365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:44:26.811097 12372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c735ea0de3547d88bf2a9f45cd4bfd8. Latest consensus state: current_term: 1 leader_uuid: "9c735ea0de3547d88bf2a9f45cd4bfd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c735ea0de3547d88bf2a9f45cd4bfd8" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 43643 } } }
I20260621 02:44:26.811070 12371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c735ea0de3547d88bf2a9f45cd4bfd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c735ea0de3547d88bf2a9f45cd4bfd8" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 43643 } } }
I20260621 02:44:26.811678 12372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:26.811678 12371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:26.812559 12379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:44:26.817111 12379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:44:26.825194 12379 catalog_manager.cc:1347] Generated new cluster ID: c68075f011f249b1955702dc5d3383a8
I20260621 02:44:26.825302 12379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:44:26.837466 12379 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:44:26.839367 12379 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:44:26.850155 12379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c735ea0de3547d88bf2a9f45cd4bfd8: Generated new TSK 0
I20260621 02:44:26.851137 12379 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
I20260621 02:44:26.891009 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:43643
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:34775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:27.201399 12391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:27.201648 12391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:27.201753 12391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:27.208612 12391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:27.208709 12391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:27.208875 12391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:44:27.218111 12391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:34775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:43643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:27.219187 12391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:27.220407 12391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:27.232796 12399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:27.232901 12396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:27.232911 12397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:27.234198 12391 server_base.cc:1034] running on GCE node
I20260621 02:44:27.234760 12391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:27.235839 12391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:27.237030 12391 hybrid_clock.cc:648] HybridClock initialized: now 1782009867236994 us; error 24 us; skew 500 ppm
I20260621 02:44:27.237350 12391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:27 dist-test-slave-hvgn krb5kdc[12285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009867, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:27.240451 12391 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:44:27.241817 12391 webserver.cc:460] Webserver started at http://127.9.166.1:37325/ using document root <none> and password file <none>
I20260621 02:44:27.242427 12391 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:27.242520 12391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:27.242772 12391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:27.245527 12391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "80091b8de81b4eb7877504d6b0509a64"
format_stamp: "Formatted at 2026-06-21 02:44:27 on dist-test-slave-hvgn"
server_key: "5eb7083d2237f909c3d2cb899f0be0fd"
server_key_iv: "ddc42cad7656932bedca70d5139f2823"
server_key_version: "encryptionkey@0"
I20260621 02:44:27.246292 12391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "80091b8de81b4eb7877504d6b0509a64"
format_stamp: "Formatted at 2026-06-21 02:44:27 on dist-test-slave-hvgn"
server_key: "5eb7083d2237f909c3d2cb899f0be0fd"
server_key_iv: "ddc42cad7656932bedca70d5139f2823"
server_key_version: "encryptionkey@0"
I20260621 02:44:27.251087 12391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260621 02:44:27.254356 12406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:27.255676 12391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:44:27.255837 12391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "80091b8de81b4eb7877504d6b0509a64"
format_stamp: "Formatted at 2026-06-21 02:44:27 on dist-test-slave-hvgn"
server_key: "5eb7083d2237f909c3d2cb899f0be0fd"
server_key_iv: "ddc42cad7656932bedca70d5139f2823"
server_key_version: "encryptionkey@0"
I20260621 02:44:27.256021 12391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:27.278753 12391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:27.282929 12391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:27.283201 12391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:27.284216 12391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:27.285462 12391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:27.285562 12391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:27.285697 12391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:27.285766 12391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:27.318113 12391 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:35537
I20260621 02:44:27.318151 12518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:35537 every 8 connection(s)
I20260621 02:44:27.319350 12391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:44:27.327724 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 12391
I20260621 02:44:27.327980 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:44:27.328572 9880 external_mini_cluster.cc:1428] Setting key 749d2217081dd323e9f8e1a3b521cad7
Jun 21 02:44:27 dist-test-slave-hvgn krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009867, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:27.340723 12521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:27.321822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:48691 (local address 127.9.166.62:43643)
0621 02:44:27.322269 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:27.322273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:27.323120 (+ 847us) server_negotiation.cc:407] Connection header received
0621 02:44:27.324949 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:27.324954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:27.325053 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:27.325168 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:27.326590 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:27.327407 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:27.329102 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:27.329663 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:27.329755 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:27.333820 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:27.333829 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:27.333837 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:27.333883 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:27.337155 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:27.338203 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:27.338207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:27.338210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:27.338355 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:27.338906 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:27.338909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:27.338912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:27.339205 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:27.339330 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:27.340299 (+ 969us) server_negotiation.cc:299] Negotiation successful
0621 02:44:27.340465 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":256,"threads_started":1}
I20260621 02:44:27.342409 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:27.322289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:43643 (local address 127.9.166.1:48691)
0621 02:44:27.322960 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:27.323001 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:27.324684 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:27.325394 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:27.325401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:27.325911 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:27.326424 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:27.326443 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:27.327552 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:27.327556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:27.328953 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:27.328964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:27.329870 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:27.329877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:27.330147 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:27.331047 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:27.331069 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:27.333577 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:27.337297 (+ 3720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:27.337309 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:27.337336 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:27.338059 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:27.338482 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:27.338491 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:27.338499 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:27.338753 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:27.339352 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:27.339363 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:27.339881 (+ 518us) client_negotiation.cc:769] Sending connection context
0621 02:44:27.340211 (+ 330us) client_negotiation.cc:240] Negotiation successful
0621 02:44:27.340828 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":258,"threads_started":1}
I20260621 02:44:27.344065 12519 heartbeater.cc:347] Connected to a master server at 127.9.166.62:43643
I20260621 02:44:27.344390 12519 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:27.345191 12519 heartbeater.cc:511] Master 127.9.166.62:43643 requested a full tablet report, sending...
I20260621 02:44:27.347270 12330 ts_manager.cc:194] Registered new tserver with Master: 80091b8de81b4eb7877504d6b0509a64 (127.9.166.1:35537)
WARNING: no policy specified for kudu/127.9.166.2@KRBTEST.COM; defaulting to no policy
I20260621 02:44:27.348850 12330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:48691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
I20260621 02:44:27.372538 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:43643
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:34775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:27.703176 12525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:27.703439 12525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:27.703605 12525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:27.710681 12525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:27.710778 12525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:27.710886 12525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:44:27.720247 12525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:34775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:43643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:27.721390 12525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:27.722615 12525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:27.735162 12530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:27.735538 12531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:27.735806 12533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:27.736464 12525 server_base.cc:1034] running on GCE node
I20260621 02:44:27.737077 12525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:27.737991 12525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:27.739223 12525 hybrid_clock.cc:648] HybridClock initialized: now 1782009867739188 us; error 47 us; skew 500 ppm
I20260621 02:44:27.739589 12525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:27 dist-test-slave-hvgn krb5kdc[12285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009867, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:27.742589 12525 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:44:27.744107 12525 webserver.cc:460] Webserver started at http://127.9.166.2:38361/ using document root <none> and password file <none>
I20260621 02:44:27.744776 12525 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:27.744918 12525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:27.745230 12525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:27.748366 12525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "d42782d917ba4076b441a1a68d3f4ac8"
format_stamp: "Formatted at 2026-06-21 02:44:27 on dist-test-slave-hvgn"
server_key: "2a87898e9df93fe1404d7cb3a8c58a0e"
server_key_iv: "57842c43bf7cfd301835745a8ac7d7d2"
server_key_version: "encryptionkey@0"
I20260621 02:44:27.749131 12525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "d42782d917ba4076b441a1a68d3f4ac8"
format_stamp: "Formatted at 2026-06-21 02:44:27 on dist-test-slave-hvgn"
server_key: "2a87898e9df93fe1404d7cb3a8c58a0e"
server_key_iv: "57842c43bf7cfd301835745a8ac7d7d2"
server_key_version: "encryptionkey@0"
I20260621 02:44:27.754189 12525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:44:27.757647 12540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:27.758919 12525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260621 02:44:27.759135 12525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "d42782d917ba4076b441a1a68d3f4ac8"
format_stamp: "Formatted at 2026-06-21 02:44:27 on dist-test-slave-hvgn"
server_key: "2a87898e9df93fe1404d7cb3a8c58a0e"
server_key_iv: "57842c43bf7cfd301835745a8ac7d7d2"
server_key_version: "encryptionkey@0"
I20260621 02:44:27.759333 12525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:27.771198 12525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:27.775238 12525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:27.775576 12525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:27.776501 12525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:27.777778 12525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:27.777889 12525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:27.778045 12525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:27.778098 12525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:27.810468 12525 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:35107
I20260621 02:44:27.810535 12652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:35107 every 8 connection(s)
I20260621 02:44:27.811996 12525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:44:27.818909 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 12525
I20260621 02:44:27.819111 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:44:27.819674 9880 external_mini_cluster.cc:1428] Setting key 00ada3a4b7d315cb6a67569982efa024
Jun 21 02:44:27 dist-test-slave-hvgn krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009867, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:27.832464 12521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:27.814598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:57715 (local address 127.9.166.62:43643)
0621 02:44:27.814805 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:27.814812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:27.815992 (+ 1180us) server_negotiation.cc:407] Connection header received
0621 02:44:27.817705 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:27.817710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:27.817776 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:27.817865 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:27.819361 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:27.820361 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:27.822009 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:27.822691 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:27.822827 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:27.826294 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:27.826302 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:27.826310 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:27.826354 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:27.829177 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:27.830074 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:27.830078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:27.830080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:27.830203 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:27.830697 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:27.830700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:27.830703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:27.831045 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:27.831198 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:27.832071 (+ 873us) server_negotiation.cc:299] Negotiation successful
0621 02:44:27.832235 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 02:44:27.834007 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:27.815091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:43643 (local address 127.9.166.2:57715)
0621 02:44:27.815841 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:27.815879 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:27.817455 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:27.818092 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:27.818101 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:27.818579 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:27.819181 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:27.819206 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:27.820541 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:27.820545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:27.821874 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:27.821886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:27.822887 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:27.822891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:27.823085 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:27.823780 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:27.823801 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:27.826083 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:27.829318 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:27.829330 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:27.829348 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:27.829955 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:27.830307 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:27.830314 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:27.830321 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:27.830566 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:27.831191 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:27.831202 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:27.831671 (+ 469us) client_negotiation.cc:769] Sending connection context
0621 02:44:27.831974 (+ 303us) client_negotiation.cc:240] Negotiation successful
0621 02:44:27.832543 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":336,"threads_started":1}
I20260621 02:44:27.835634 12653 heartbeater.cc:347] Connected to a master server at 127.9.166.62:43643
I20260621 02:44:27.835929 12653 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:27.836683 12653 heartbeater.cc:511] Master 127.9.166.62:43643 requested a full tablet report, sending...
I20260621 02:44:27.838131 12330 ts_manager.cc:194] Registered new tserver with Master: d42782d917ba4076b441a1a68d3f4ac8 (127.9.166.2:35107)
WARNING: no policy specified for kudu/127.9.166.3@KRBTEST.COM; defaulting to no policy
I20260621 02:44:27.839049 12330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:57715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
I20260621 02:44:27.862474 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:43643
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:34775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:28.177851 12658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:28.178129 12658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:28.178220 12658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:28.185073 12658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:28.185171 12658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:28.185283 12658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:44:28.194448 12658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:34775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:43643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:28.195647 12658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:28.196825 12658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:28.208900 12663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:28.208931 12664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:28.209436 12666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:28.210139 12658 server_base.cc:1034] running on GCE node
I20260621 02:44:28.210707 12658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:28.211606 12658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:28.212837 12658 hybrid_clock.cc:648] HybridClock initialized: now 1782009868212798 us; error 25 us; skew 500 ppm
I20260621 02:44:28.213140 12658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:28 dist-test-slave-hvgn krb5kdc[12285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009868, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:28.216406 12658 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:44:28.217693 12658 webserver.cc:460] Webserver started at http://127.9.166.3:39149/ using document root <none> and password file <none>
I20260621 02:44:28.218349 12658 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:28.218495 12658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:28.218797 12658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:28.221560 12658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "f15d0ba710804f1c8cb1e8882d08c0da"
format_stamp: "Formatted at 2026-06-21 02:44:28 on dist-test-slave-hvgn"
server_key: "74a19aec6e71a3ca344930cc4d20dba0"
server_key_iv: "dc80ee0dbd64bb7fde6948a4adffc84f"
server_key_version: "encryptionkey@0"
I20260621 02:44:28.222302 12658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "f15d0ba710804f1c8cb1e8882d08c0da"
format_stamp: "Formatted at 2026-06-21 02:44:28 on dist-test-slave-hvgn"
server_key: "74a19aec6e71a3ca344930cc4d20dba0"
server_key_iv: "dc80ee0dbd64bb7fde6948a4adffc84f"
server_key_version: "encryptionkey@0"
I20260621 02:44:28.227057 12658 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260621 02:44:28.230333 12673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:28.231688 12658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:44:28.231856 12658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "f15d0ba710804f1c8cb1e8882d08c0da"
format_stamp: "Formatted at 2026-06-21 02:44:28 on dist-test-slave-hvgn"
server_key: "74a19aec6e71a3ca344930cc4d20dba0"
server_key_iv: "dc80ee0dbd64bb7fde6948a4adffc84f"
server_key_version: "encryptionkey@0"
I20260621 02:44:28.232044 12658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:28.250245 12658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:28.254412 12658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:28.254693 12658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:28.255631 12658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:28.256954 12658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:28.257052 12658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:28.257211 12658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:28.257283 12658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:28.290150 12658 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:38991
I20260621 02:44:28.290197 12785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:38991 every 8 connection(s)
I20260621 02:44:28.291414 12658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:44:28.299157 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 12658
I20260621 02:44:28.299350 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:44:28.299894 9880 external_mini_cluster.cc:1428] Setting key 5e8bb0c6445b89e01e631ae6670af18a
Jun 21 02:44:28 dist-test-slave-hvgn krb5kdc[12285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009868, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:28.310390 12521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:28.293879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:36919 (local address 127.9.166.62:43643)
0621 02:44:28.294104 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:28.294110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:28.295227 (+ 1117us) server_negotiation.cc:407] Connection header received
0621 02:44:28.296936 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:28.296940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:28.297006 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:28.297098 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:28.298464 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:28.299179 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:28.301369 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:28.301953 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:28.302077 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:28.305423 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:28.305427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:28.305430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:28.305459 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:28.307909 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:28.308550 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:28.308554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:28.308556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:28.308737 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:28.309070 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:28.309074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:28.309077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:28.309315 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:28.309418 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:28.310006 (+ 588us) server_negotiation.cc:299] Negotiation successful
0621 02:44:28.310157 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 02:44:28.311205 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:28.294378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:43643 (local address 127.9.166.3:36919)
0621 02:44:28.295065 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:28.295106 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:28.296696 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:28.297309 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:28.297316 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:28.297791 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:28.298326 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:28.298342 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:28.299368 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:28.299374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:28.301245 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:28.301260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:28.302153 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:28.302158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:28.302336 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:28.303042 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:28.303076 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:28.305264 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:28.308021 (+ 2757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:28.308027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:28.308037 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:28.308453 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:28.308821 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:28.308824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:28.308826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:28.308977 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:28.309411 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:28.309416 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:28.309688 (+ 272us) client_negotiation.cc:769] Sending connection context
0621 02:44:28.309910 (+ 222us) client_negotiation.cc:240] Negotiation successful
0621 02:44:28.310232 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":284,"threads_started":1}
I20260621 02:44:28.312671 12786 heartbeater.cc:347] Connected to a master server at 127.9.166.62:43643
I20260621 02:44:28.312947 12786 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:28.313760 12786 heartbeater.cc:511] Master 127.9.166.62:43643 requested a full tablet report, sending...
I20260621 02:44:28.315275 12330 ts_manager.cc:194] Registered new tserver with Master: f15d0ba710804f1c8cb1e8882d08c0da (127.9.166.3:38991)
I20260621 02:44:28.316064 12330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:36919
I20260621 02:44:28.326949 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:44:28.351900 12519 heartbeater.cc:503] Master 127.9.166.62:43643 was elected leader, sending a full tablet report...
W20260621 02:44:28.356419 12521 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:44:28.354091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38228 (local address 127.9.166.62:43643)
0621 02:44:28.354274 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:28.354278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:28.354759 (+ 481us) server_negotiation.cc:407] Connection header received
0621 02:44:28.354888 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:28.354891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:28.354947 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:28.355029 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:28.356192 (+ 1163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38228: BlockingRecv error: recv got EOF from 127.0.0.1:38228 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:44:28.360329 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 12391
W20260621 02:44:28.363025 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 12391: Not authorized: Client connection negotiation failed: client connection to 127.9.166.1:35537: server requires authentication, but client does not have Kerberos credentials available
I20260621 02:44:28.363305 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 12391
I20260621 02:44:28.379040 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 12525
W20260621 02:44:28.381609 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 12525: Not authorized: Client connection negotiation failed: client connection to 127.9.166.2:35107: server requires authentication, but client does not have Kerberos credentials available
I20260621 02:44:28.381794 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 12525
I20260621 02:44:28.396694 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 12658
W20260621 02:44:28.399489 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 12658: Not authorized: Client connection negotiation failed: client connection to 127.9.166.3:38991: server requires authentication, but client does not have Kerberos credentials available
I20260621 02:44:28.399631 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 12658
I20260621 02:44:28.414446 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 12298
I20260621 02:44:28.415567 12360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:28.516538 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 12298
2026-06-21T02:44:28Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3410 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:44:28 dist-test-slave-hvgn krb5kdc[12803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:44:28 dist-test-slave-hvgn krb5kdc[12803](info): set up 2 sockets
Jun 21 02:44:28 dist-test-slave-hvgn krb5kdc[12803](info): commencing operation
krb5kdc: starting...
W20260621 02:44:29.611665 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:44:29 dist-test-slave-hvgn krb5kdc[12803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:44:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:44:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:44:29.791256 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:36457
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:36457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:30.106197 12816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:30.106539 12816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:30.106703 12816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:30.113961 12816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:44:30.114120 12816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:30.114233 12816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:30.114333 12816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:44:30.114413 12816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:44:30.124161 12816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:36457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:36457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:30.125463 12816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:30.126780 12816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:30.138134 12821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:30.138132 12822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:30.138214 12824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:30.139070 12816 server_base.cc:1034] running on GCE node
I20260621 02:44:30.139863 12816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:30.141155 12816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:30.142436 12816 hybrid_clock.cc:648] HybridClock initialized: now 1782009870142379 us; error 47 us; skew 500 ppm
I20260621 02:44:30.142822 12816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:30 dist-test-slave-hvgn krb5kdc[12803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009870, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:30.145961 12816 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:44:30.147549 12816 webserver.cc:460] Webserver started at http://127.9.166.62:37531/ using document root <none> and password file <none>
I20260621 02:44:30.148214 12816 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:30.148361 12816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:30.148665 12816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:30.151538 12816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "6f169fa0810742628b2e2c8cc699b9e4"
format_stamp: "Formatted at 2026-06-21 02:44:30 on dist-test-slave-hvgn"
server_key: "7f95276f7fa5d0ed7f931b2d80a49e47"
server_key_iv: "a47130f1c211167c6f19e5ba40e9d2f1"
server_key_version: "encryptionkey@0"
I20260621 02:44:30.152315 12816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "6f169fa0810742628b2e2c8cc699b9e4"
format_stamp: "Formatted at 2026-06-21 02:44:30 on dist-test-slave-hvgn"
server_key: "7f95276f7fa5d0ed7f931b2d80a49e47"
server_key_iv: "a47130f1c211167c6f19e5ba40e9d2f1"
server_key_version: "encryptionkey@0"
I20260621 02:44:30.157225 12816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:44:30.160795 12831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:30.162110 12816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:44:30.162312 12816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "6f169fa0810742628b2e2c8cc699b9e4"
format_stamp: "Formatted at 2026-06-21 02:44:30 on dist-test-slave-hvgn"
server_key: "7f95276f7fa5d0ed7f931b2d80a49e47"
server_key_iv: "a47130f1c211167c6f19e5ba40e9d2f1"
server_key_version: "encryptionkey@0"
I20260621 02:44:30.162521 12816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:30.188498 12816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:30.192669 12816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:30.192960 12816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:30.210650 12816 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:36457
I20260621 02:44:30.210688 12882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:36457 every 8 connection(s)
I20260621 02:44:30.212054 12816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:44:30.215618 12883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:30.218928 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 12816
I20260621 02:44:30.219120 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:44:30.219949 9880 external_mini_cluster.cc:1428] Setting key 55bf0d45558ffac755b93107aa8eb46d
I20260621 02:44:30.228242 12883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4: Bootstrap starting.
I20260621 02:44:30.232333 12883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:30.233738 12883 log.cc:824] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4: Log is configured to *not* fsync() on all Append() calls
Jun 21 02:44:30 dist-test-slave-hvgn krb5kdc[12803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:30.236937 12883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4: No bootstrap required, opened a new log
I20260621 02:44:30.243646 12886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:30.221798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43340 (local address 127.9.166.62:36457)
0621 02:44:30.222537 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:30.222548 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:30.222586 (+ 38us) server_negotiation.cc:407] Connection header received
0621 02:44:30.224223 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:30.224244 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:30.224619 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:30.224921 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:30.226728 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:30.228076 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:30.229818 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:30.230495 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:30.230626 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:30.236158 (+ 5532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:30.236165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:30.236178 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:30.236210 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:30.239358 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:30.240243 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:30.240248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:30.240255 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:30.240459 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:30.240880 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:30.240886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:30.240888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:30.241389 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:30.241635 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:30.242160 (+ 525us) server_negotiation.cc:299] Negotiation successful
0621 02:44:30.242592 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":345,"threads_started":1}
I20260621 02:44:30.246749 12883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f169fa0810742628b2e2c8cc699b9e4" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36457 } }
I20260621 02:44:30.247187 12883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:30.247319 12883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f169fa0810742628b2e2c8cc699b9e4, State: Initialized, Role: FOLLOWER
I20260621 02:44:30.247699 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:44:30.248006 12883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f169fa0810742628b2e2c8cc699b9e4" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36457 } }
I20260621 02:44:30.248240 12883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:44:30.248371 12883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:44:30.248556 12883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:30.250737 12883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f169fa0810742628b2e2c8cc699b9e4" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36457 } }
I20260621 02:44:30.251223 12883 leader_election.cc:302] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f169fa0810742628b2e2c8cc699b9e4; no voters:
I20260621 02:44:30.251848 12883 leader_election.cc:288] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:44:30.251984 12888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:30.253785 12883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:44:30.258741 12888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4 [term 1 LEADER]: Becoming Leader. State: Replica: 6f169fa0810742628b2e2c8cc699b9e4, State: Running, Role: LEADER
I20260621 02:44:30.259377 12888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f169fa0810742628b2e2c8cc699b9e4" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36457 } }
I20260621 02:44:30.262220 12889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f169fa0810742628b2e2c8cc699b9e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f169fa0810742628b2e2c8cc699b9e4" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36457 } } }
I20260621 02:44:30.262560 12889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:30.263039 12890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f169fa0810742628b2e2c8cc699b9e4. Latest consensus state: current_term: 1 leader_uuid: "6f169fa0810742628b2e2c8cc699b9e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f169fa0810742628b2e2c8cc699b9e4" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36457 } } }
I20260621 02:44:30.263334 12890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:30.275815 12899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:44:30.280800 12899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:44:30.290145 12899 catalog_manager.cc:1347] Generated new cluster ID: 26e7f380f7cd450eb47fe1174f160c4f
I20260621 02:44:30.290299 12899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:44:30.300944 12886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:30.293447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43354 (local address 127.9.166.62:36457)
0621 02:44:30.293622 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:30.293628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:30.293717 (+ 89us) server_negotiation.cc:407] Connection header received
0621 02:44:30.293990 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:30.293996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:30.294078 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:30.294201 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:30.295199 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:30.296118 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:30.297505 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:30.298023 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:30.298165 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:30.299331 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:30.299344 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:30.299348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:44:30.300021 (+ 673us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:30.300081 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:44:30.300108 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:30.300235 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:30.300392 (+ 157us) server_negotiation.cc:299] Negotiation successful
0621 02:44:30.300494 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 02:44:30.313506 12899 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:44:30.314894 12899 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:44:30.326146 12899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f169fa0810742628b2e2c8cc699b9e4: Generated new TSK 0
I20260621 02:44:30.327056 12899 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:44:30.338472 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 12816
I20260621 02:44:30.340046 12878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:30.431195 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 12816
2026-06-21T02:44:30Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1909 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:44:30 dist-test-slave-hvgn krb5kdc[12918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:44:30 dist-test-slave-hvgn krb5kdc[12918](info): set up 2 sockets
Jun 21 02:44:30 dist-test-slave-hvgn krb5kdc[12918](info): commencing operation
krb5kdc: starting...
W20260621 02:44:31.511808 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:44:31 dist-test-slave-hvgn krb5kdc[12918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:44:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:44:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:44:31.694203 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:39321
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:39321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:32.000995 12931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:32.001253 12931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:32.001417 12931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:32.008278 12931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:44:32.008369 12931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:32.008484 12931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:32.008585 12931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:44:32.008672 12931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:44:32.017782 12931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:39321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:39321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:32.018997 12931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:32.020248 12931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:32.031777 12936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:32.031832 12939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:32.031747 12937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:32.032691 12931 server_base.cc:1034] running on GCE node
I20260621 02:44:32.033412 12931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:32.034624 12931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:32.035822 12931 hybrid_clock.cc:648] HybridClock initialized: now 1782009872035779 us; error 36 us; skew 500 ppm
I20260621 02:44:32.036267 12931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:32 dist-test-slave-hvgn krb5kdc[12918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009872, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:32.039199 12931 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:44:32.040567 12931 webserver.cc:460] Webserver started at http://127.9.166.62:41859/ using document root <none> and password file <none>
I20260621 02:44:32.041144 12931 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:32.041239 12931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:32.041479 12931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:32.044294 12931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "abb5b68feecf41b792b2cea197ab2483"
format_stamp: "Formatted at 2026-06-21 02:44:32 on dist-test-slave-hvgn"
server_key: "3f01947e35a9e021f76ed0f4e724e04a"
server_key_iv: "b68140c8655ef123c8003b5ea690fbc6"
server_key_version: "encryptionkey@0"
I20260621 02:44:32.044986 12931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "abb5b68feecf41b792b2cea197ab2483"
format_stamp: "Formatted at 2026-06-21 02:44:32 on dist-test-slave-hvgn"
server_key: "3f01947e35a9e021f76ed0f4e724e04a"
server_key_iv: "b68140c8655ef123c8003b5ea690fbc6"
server_key_version: "encryptionkey@0"
I20260621 02:44:32.049659 12931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260621 02:44:32.052752 12946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:32.053922 12931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260621 02:44:32.054082 12931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "abb5b68feecf41b792b2cea197ab2483"
format_stamp: "Formatted at 2026-06-21 02:44:32 on dist-test-slave-hvgn"
server_key: "3f01947e35a9e021f76ed0f4e724e04a"
server_key_iv: "b68140c8655ef123c8003b5ea690fbc6"
server_key_version: "encryptionkey@0"
I20260621 02:44:32.054265 12931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:32.070390 12931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:32.073962 12931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:32.074229 12931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:32.091452 12931 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:39321
I20260621 02:44:32.091542 12997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:39321 every 8 connection(s)
I20260621 02:44:32.092753 12931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:44:32.096575 12998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:32.100526 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 12931
I20260621 02:44:32.100745 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:44:32.101544 9880 external_mini_cluster.cc:1428] Setting key 152bbe541f83ca0bdd44fadecd0eca60
I20260621 02:44:32.108006 12998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483: Bootstrap starting.
I20260621 02:44:32.111598 12998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:32.112761 12998 log.cc:824] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:32.116185 12998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483: No bootstrap required, opened a new log
Jun 21 02:44:32 dist-test-slave-hvgn krb5kdc[12918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:32.123665 12998 raft_consensus.cc:348] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb5b68feecf41b792b2cea197ab2483" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39321 } }
I20260621 02:44:32.124185 12998 raft_consensus.cc:374] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:32.124307 12998 raft_consensus.cc:729] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb5b68feecf41b792b2cea197ab2483, State: Initialized, Role: FOLLOWER
I20260621 02:44:32.124853 12998 consensus_queue.cc:260] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb5b68feecf41b792b2cea197ab2483" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39321 } }
I20260621 02:44:32.125084 12998 raft_consensus.cc:388] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:44:32.125228 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:32.103597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38346 (local address 127.9.166.62:39321)
0621 02:44:32.104338 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:32.104349 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:32.104386 (+ 37us) server_negotiation.cc:407] Connection header received
0621 02:44:32.106550 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:32.106572 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:32.106941 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:32.107271 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:32.109006 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:32.110463 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:32.112255 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:32.113010 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:32.113143 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:32.117771 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:32.117778 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:32.117792 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:32.117828 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:32.121031 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:32.121712 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:32.121718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:32.121725 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:32.121882 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:32.122295 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:32.122298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:32.122300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:32.122785 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:32.123116 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:32.123547 (+ 431us) server_negotiation.cc:299] Negotiation successful
0621 02:44:32.123934 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":327,"threads_started":1}
I20260621 02:44:32.125248 12998 raft_consensus.cc:482] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:44:32.125466 12998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:32.128414 12998 raft_consensus.cc:504] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb5b68feecf41b792b2cea197ab2483" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39321 } }
I20260621 02:44:32.128892 12998 leader_election.cc:302] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abb5b68feecf41b792b2cea197ab2483; no voters:
I20260621 02:44:32.129429 12998 leader_election.cc:288] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:44:32.129571 13003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:32.130230 13003 raft_consensus.cc:686] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483 [term 1 LEADER]: Becoming Leader. State: Replica: abb5b68feecf41b792b2cea197ab2483, State: Running, Role: LEADER
I20260621 02:44:32.130708 13003 consensus_queue.cc:237] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb5b68feecf41b792b2cea197ab2483" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39321 } }
I20260621 02:44:32.131350 12998 sys_catalog.cc:564] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:44:32.133414 13005 sys_catalog.cc:455] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abb5b68feecf41b792b2cea197ab2483. Latest consensus state: current_term: 1 leader_uuid: "abb5b68feecf41b792b2cea197ab2483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb5b68feecf41b792b2cea197ab2483" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39321 } } }
I20260621 02:44:32.133771 13005 sys_catalog.cc:458] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:32.134892 13011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:44:32.134658 13004 sys_catalog.cc:455] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abb5b68feecf41b792b2cea197ab2483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb5b68feecf41b792b2cea197ab2483" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39321 } } }
I20260621 02:44:32.136018 13004 sys_catalog.cc:458] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:32.142927 13011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:44:32.151460 13011 catalog_manager.cc:1347] Generated new cluster ID: 878662814c8a49da9fa0434f59706a1f
I20260621 02:44:32.151564 13011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:44:32.173566 13011 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:44:32.174801 13011 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:44:32.184290 13011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abb5b68feecf41b792b2cea197ab2483: Generated new TSK 0
I20260621 02:44:32.185194 13011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
I20260621 02:44:32.230409 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:39321
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:32.559553 13024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:32.559806 13024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:32.559899 13024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:32.566810 13024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:32.566906 13024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:32.567014 13024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:44:32.576330 13024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:39321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:32.577606 13024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:32.578830 13024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:32.591259 13032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:32.591312 13029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:32.591563 13030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:32.592221 13024 server_base.cc:1034] running on GCE node
I20260621 02:44:32.592777 13024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:32.593626 13024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:32.594825 13024 hybrid_clock.cc:648] HybridClock initialized: now 1782009872594785 us; error 31 us; skew 500 ppm
I20260621 02:44:32.595156 13024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:32 dist-test-slave-hvgn krb5kdc[12918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009872, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:32.598307 13024 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:44:32.599728 13024 webserver.cc:460] Webserver started at http://127.9.166.1:35537/ using document root <none> and password file <none>
I20260621 02:44:32.600387 13024 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:32.600556 13024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:32.600845 13024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:32.603695 13024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "c661bf14ed6740d1b74888ced1306aa0"
format_stamp: "Formatted at 2026-06-21 02:44:32 on dist-test-slave-hvgn"
server_key: "e3acefa4ee35c0f1dd4558aaad5b617c"
server_key_iv: "37a7d93d5dcbd487d7f184184d46c2e0"
server_key_version: "encryptionkey@0"
I20260621 02:44:32.604486 13024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "c661bf14ed6740d1b74888ced1306aa0"
format_stamp: "Formatted at 2026-06-21 02:44:32 on dist-test-slave-hvgn"
server_key: "e3acefa4ee35c0f1dd4558aaad5b617c"
server_key_iv: "37a7d93d5dcbd487d7f184184d46c2e0"
server_key_version: "encryptionkey@0"
I20260621 02:44:32.609436 13024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:44:32.612736 13039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:32.614004 13024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:44:32.614179 13024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "c661bf14ed6740d1b74888ced1306aa0"
format_stamp: "Formatted at 2026-06-21 02:44:32 on dist-test-slave-hvgn"
server_key: "e3acefa4ee35c0f1dd4558aaad5b617c"
server_key_iv: "37a7d93d5dcbd487d7f184184d46c2e0"
server_key_version: "encryptionkey@0"
I20260621 02:44:32.614374 13024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:32.645530 13024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:32.649376 13024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:32.649665 13024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:32.650557 13024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:32.651896 13024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:32.651999 13024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:32.652141 13024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:32.652220 13024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:32.682431 13024 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:46863
I20260621 02:44:32.682468 13151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:46863 every 8 connection(s)
I20260621 02:44:32.683761 13024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:44:32.689666 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 13024
I20260621 02:44:32.689868 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:44:32.690500 9880 external_mini_cluster.cc:1428] Setting key c986c58ec41feadbf76f728087714b56
Jun 21 02:44:32 dist-test-slave-hvgn krb5kdc[12918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009872, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:32.703959 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:32.686423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:58531 (local address 127.9.166.62:39321)
0621 02:44:32.686885 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:32.686891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:32.687864 (+ 973us) server_negotiation.cc:407] Connection header received
0621 02:44:32.689626 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:32.689633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:32.689735 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:32.689880 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:32.691627 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:32.692567 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:32.694115 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:32.694840 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:32.694989 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:32.698482 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:32.698488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:32.698491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:32.698520 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:32.701002 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:32.701705 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:32.701714 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:32.701717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:32.701828 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:32.702307 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:32.702318 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:32.702320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:32.702566 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:32.702707 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:32.703359 (+ 652us) server_negotiation.cc:299] Negotiation successful
0621 02:44:32.703592 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":234,"threads_started":1}
I20260621 02:44:32.704873 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:32.686992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39321 (local address 127.9.166.1:58531)
0621 02:44:32.687711 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:32.687755 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:32.689350 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:32.690133 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:32.690141 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:32.690705 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:32.691432 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:32.691464 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:32.692730 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:32.692734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:32.693962 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:32.693973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:32.695054 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:32.695058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:32.695248 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:32.695969 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:32.695990 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:32.698272 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:32.701142 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:32.701151 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:32.701166 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:32.701573 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:32.701964 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:32.701970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:32.701974 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:32.702189 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:32.702696 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:32.702703 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:32.703018 (+ 315us) client_negotiation.cc:769] Sending connection context
0621 02:44:32.703263 (+ 245us) client_negotiation.cc:240] Negotiation successful
0621 02:44:32.703795 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":300,"threads_started":1}
I20260621 02:44:32.706578 13152 heartbeater.cc:347] Connected to a master server at 127.9.166.62:39321
I20260621 02:44:32.706859 13152 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:32.707638 13152 heartbeater.cc:511] Master 127.9.166.62:39321 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.166.2@KRBTEST.COM; defaulting to no policy
I20260621 02:44:32.709841 12963 ts_manager.cc:194] Registered new tserver with Master: c661bf14ed6740d1b74888ced1306aa0 (127.9.166.1:46863)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.2@KRBTEST.COM" created.
I20260621 02:44:32.711498 12963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:58531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
I20260621 02:44:32.732342 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:39321
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:33.039798 13158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:33.040061 13158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:33.040153 13158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:33.046999 13158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:33.047091 13158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:33.047196 13158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:44:33.056455 13158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:39321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:33.057647 13158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:33.058917 13158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:33.071635 13164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:33.071771 13163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:33.071609 13166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:33.072214 13158 server_base.cc:1034] running on GCE node
I20260621 02:44:33.072878 13158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:33.073760 13158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:33.074955 13158 hybrid_clock.cc:648] HybridClock initialized: now 1782009873074910 us; error 29 us; skew 500 ppm
I20260621 02:44:33.075249 13158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:33 dist-test-slave-hvgn krb5kdc[12918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009873, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:33.078251 13158 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:44:33.079690 13158 webserver.cc:460] Webserver started at http://127.9.166.2:40993/ using document root <none> and password file <none>
I20260621 02:44:33.080305 13158 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:33.080408 13158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:33.080638 13158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:33.083338 13158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "6e14373bb45f478f80fe38c43bb9910a"
format_stamp: "Formatted at 2026-06-21 02:44:33 on dist-test-slave-hvgn"
server_key: "1f2bb3416ef0c9682f0995c3272b5d75"
server_key_iv: "99becdfed9897081ea001f4a5aee984e"
server_key_version: "encryptionkey@0"
I20260621 02:44:33.084124 13158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "6e14373bb45f478f80fe38c43bb9910a"
format_stamp: "Formatted at 2026-06-21 02:44:33 on dist-test-slave-hvgn"
server_key: "1f2bb3416ef0c9682f0995c3272b5d75"
server_key_iv: "99becdfed9897081ea001f4a5aee984e"
server_key_version: "encryptionkey@0"
I20260621 02:44:33.089059 13158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260621 02:44:33.092391 13173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:33.093752 13158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:44:33.093938 13158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "6e14373bb45f478f80fe38c43bb9910a"
format_stamp: "Formatted at 2026-06-21 02:44:33 on dist-test-slave-hvgn"
server_key: "1f2bb3416ef0c9682f0995c3272b5d75"
server_key_iv: "99becdfed9897081ea001f4a5aee984e"
server_key_version: "encryptionkey@0"
I20260621 02:44:33.094126 13158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:33.121769 13158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:33.126068 13158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:33.126370 13158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:33.127348 13158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:33.128715 13158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:33.128820 13158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:33.128973 13158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:33.129045 13158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:33.163749 13158 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:45513
I20260621 02:44:33.163834 13285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:45513 every 8 connection(s)
I20260621 02:44:33.164999 13158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:44:33.168674 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 13158
I20260621 02:44:33.168849 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:44:33.169335 9880 external_mini_cluster.cc:1428] Setting key 3501996b44dae3420523bfe90d01775f
Jun 21 02:44:33 dist-test-slave-hvgn krb5kdc[12918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009873, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:33.186743 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:33.167571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:44373 (local address 127.9.166.62:39321)
0621 02:44:33.167749 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:33.167754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:33.168929 (+ 1175us) server_negotiation.cc:407] Connection header received
0621 02:44:33.171159 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:33.171164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:33.171254 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:33.171333 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:33.172788 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:33.173978 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:33.175460 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:33.176195 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:33.176331 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:33.179870 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:33.179879 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:33.179887 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:33.179938 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:33.183568 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:33.184212 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:33.184220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:33.184227 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:33.184436 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:33.184918 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:33.184927 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:33.184933 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:33.185317 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:33.185481 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:33.186077 (+ 596us) server_negotiation.cc:299] Negotiation successful
0621 02:44:33.186333 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:44:33.187526 13288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:33.168003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39321 (local address 127.9.166.2:44373)
0621 02:44:33.168736 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:33.168814 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:33.170869 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:33.171605 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:33.171612 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:33.172100 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:33.172621 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:33.172636 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:33.174130 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:33.174134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:33.175295 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:33.175306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:33.176406 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:33.176411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:33.176653 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:33.177322 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:33.177342 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:33.179653 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:33.183689 (+ 4036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:33.183695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:33.183706 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:33.184080 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:33.184558 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:33.184562 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:33.184565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:33.184780 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:33.185447 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:33.185453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:33.185716 (+ 263us) client_negotiation.cc:769] Sending connection context
0621 02:44:33.185921 (+ 205us) client_negotiation.cc:240] Negotiation successful
0621 02:44:33.186349 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":318,"threads_started":1}
WARNING: no policy specified for kudu/127.9.166.3@KRBTEST.COM; defaulting to no policy
I20260621 02:44:33.188958 13286 heartbeater.cc:347] Connected to a master server at 127.9.166.62:39321
I20260621 02:44:33.189267 13286 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:33.189965 13286 heartbeater.cc:511] Master 127.9.166.62:39321 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.3@KRBTEST.COM" created.
I20260621 02:44:33.191552 12963 ts_manager.cc:194] Registered new tserver with Master: 6e14373bb45f478f80fe38c43bb9910a (127.9.166.2:45513)
I20260621 02:44:33.192809 12963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:44373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
I20260621 02:44:33.212927 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:39321
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:33.525197 13291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:33.525463 13291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:33.525609 13291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:33.532771 13291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:33.532886 13291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:33.533066 13291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:44:33.542591 13291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:39321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:33.543852 13291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:33.545141 13291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:33.558038 13296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:33.558063 13297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:33.558619 13299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:33.559090 13291 server_base.cc:1034] running on GCE node
I20260621 02:44:33.559695 13291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:33.560534 13291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:33.561751 13291 hybrid_clock.cc:648] HybridClock initialized: now 1782009873561697 us; error 42 us; skew 500 ppm
I20260621 02:44:33.562063 13291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:33 dist-test-slave-hvgn krb5kdc[12918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009873, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:33.565377 13291 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:44:33.566764 13291 webserver.cc:460] Webserver started at http://127.9.166.3:38789/ using document root <none> and password file <none>
I20260621 02:44:33.567487 13291 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:33.567591 13291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:33.567832 13291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:33.570652 13291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "3831d0c3ce95472e90fb445fde9a7463"
format_stamp: "Formatted at 2026-06-21 02:44:33 on dist-test-slave-hvgn"
server_key: "0010124936389dcbda46b57e6ebb8134"
server_key_iv: "36e7f418bc9d482eee2df8d7f388149a"
server_key_version: "encryptionkey@0"
I20260621 02:44:33.571477 13291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "3831d0c3ce95472e90fb445fde9a7463"
format_stamp: "Formatted at 2026-06-21 02:44:33 on dist-test-slave-hvgn"
server_key: "0010124936389dcbda46b57e6ebb8134"
server_key_iv: "36e7f418bc9d482eee2df8d7f388149a"
server_key_version: "encryptionkey@0"
I20260621 02:44:33.576490 13291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:44:33.579846 13306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:33.581174 13291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260621 02:44:33.581336 13291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "3831d0c3ce95472e90fb445fde9a7463"
format_stamp: "Formatted at 2026-06-21 02:44:33 on dist-test-slave-hvgn"
server_key: "0010124936389dcbda46b57e6ebb8134"
server_key_iv: "36e7f418bc9d482eee2df8d7f388149a"
server_key_version: "encryptionkey@0"
I20260621 02:44:33.581527 13291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:33.604173 13291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:33.607882 13291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:33.608176 13291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:33.609088 13291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:33.610348 13291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:33.610422 13291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:33.610587 13291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:33.610637 13291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:33.644680 13291 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:40267
I20260621 02:44:33.644774 13418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:40267 every 8 connection(s)
I20260621 02:44:33.645925 13291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:44:33.649482 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 13291
I20260621 02:44:33.649679 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:44:33.650194 9880 external_mini_cluster.cc:1428] Setting key 2a3a38631c12b7e1f06c9f544491ab1e
Jun 21 02:44:33 dist-test-slave-hvgn krb5kdc[12918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009873, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:33.665366 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:33.648462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:46675 (local address 127.9.166.62:39321)
0621 02:44:33.648644 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:33.648649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:33.649937 (+ 1288us) server_negotiation.cc:407] Connection header received
0621 02:44:33.651984 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:33.651989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:33.652056 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:33.652138 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:33.653699 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:33.654444 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:33.655989 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:33.656388 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:33.656480 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:33.659814 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:33.659819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:33.659823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:33.659852 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:33.662190 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:33.662831 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:33.662835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:33.662837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:33.662943 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:33.663443 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:33.663460 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:33.663462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:33.663799 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:33.663929 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:33.664827 (+ 898us) server_negotiation.cc:299] Negotiation successful
0621 02:44:33.665004 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:44:33.666317 13421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:33.648933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39321 (local address 127.9.166.3:46675)
0621 02:44:33.649768 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:33.649818 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:33.651725 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:33.652384 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:33.652392 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:33.652991 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:33.653548 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:33.653564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:33.654582 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:33.654586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:33.655869 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:33.655880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:33.656568 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:33.656573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:33.656757 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:33.657343 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:33.657363 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:33.659642 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:33.662308 (+ 2666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:33.662313 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:33.662324 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:33.662731 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:33.663061 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:33.663065 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:33.663067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:33.663237 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:33.663958 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:33.663966 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:33.664351 (+ 385us) client_negotiation.cc:769] Sending connection context
0621 02:44:33.664675 (+ 324us) client_negotiation.cc:240] Negotiation successful
0621 02:44:33.665181 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":348,"threads_started":1}
I20260621 02:44:33.667637 13419 heartbeater.cc:347] Connected to a master server at 127.9.166.62:39321
I20260621 02:44:33.667891 13419 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:33.668514 13419 heartbeater.cc:511] Master 127.9.166.62:39321 requested a full tablet report, sending...
I20260621 02:44:33.669802 12963 ts_manager.cc:194] Registered new tserver with Master: 3831d0c3ce95472e90fb445fde9a7463 (127.9.166.3:40267)
I20260621 02:44:33.670542 12963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:46675
I20260621 02:44:33.677662 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 02:44:33 dist-test-slave-hvgn krb5kdc[12918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009873, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
I20260621 02:44:33.715065 13152 heartbeater.cc:503] Master 127.9.166.62:39321 was elected leader, sending a full tablet report...
Jun 21 02:44:33 dist-test-slave-hvgn krb5kdc[12918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009873, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:33.726209 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:33.712383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34530 (local address 127.9.166.62:39321)
0621 02:44:33.712607 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:33.712615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:33.713309 (+ 694us) server_negotiation.cc:407] Connection header received
0621 02:44:33.713526 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:33.713530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:33.713603 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:33.713691 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:33.715102 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:33.716024 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:33.717399 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:33.717819 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:33.717909 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:33.721567 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:33.721585 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:33.721589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:33.721619 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:33.723990 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:33.724554 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:33.724558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:33.724560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:33.724697 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:33.725063 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:33.725066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:33.725068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:33.725360 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:33.725535 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:33.725776 (+ 241us) server_negotiation.cc:299] Negotiation successful
0621 02:44:33.725926 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260621 02:44:33.726827 12963 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34530
I20260621 02:44:33.729008 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 13024
Jun 21 02:44:33 dist-test-slave-hvgn krb5kdc[12918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009873, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:44:33.744624 13430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:33.730219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.9.166.1:46863)
0621 02:44:33.730828 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:33.730835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:33.730860 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:44:33.730914 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:33.730920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:33.731151 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:33.731276 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:33.732563 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:33.733624 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:33.734804 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:33.735532 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:33.735650 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:33.739454 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:33.739465 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:33.739479 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:33.739511 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:33.742266 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:33.742878 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:33.742886 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:33.742888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:33.742997 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:33.743301 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:33.743304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:33.743306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:33.743684 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:33.743900 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:33.744172 (+ 272us) server_negotiation.cc:299] Negotiation successful
0621 02:44:33.744352 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":308,"threads_started":1}
W20260621 02:44:33.745837 13147 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47516
W20260621 02:44:33.746958 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 13024: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:44:33.747181 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 13024
I20260621 02:44:33.762903 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 13158
Jun 21 02:44:33 dist-test-slave-hvgn krb5kdc[12918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009873, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.166.2@KRBTEST.COM
I20260621 02:44:33.778414 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:33.763770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49864 (local address 127.9.166.2:45513)
0621 02:44:33.764354 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:33.764361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:33.764386 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:44:33.764444 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:33.764450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:33.764653 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:33.764776 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:33.765884 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:33.766944 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:33.768247 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:33.768759 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:33.768855 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:33.772697 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:33.772708 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:33.772721 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:33.772762 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:33.775643 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:33.776350 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:33.776357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:33.776359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:33.776478 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:33.776845 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:33.776855 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:33.776857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:33.777183 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:33.777491 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:33.777824 (+ 333us) server_negotiation.cc:299] Negotiation successful
0621 02:44:33.778088 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":325,"threads_started":1}
W20260621 02:44:33.779695 13281 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49864
W20260621 02:44:33.780771 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 13158: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:44:33.780905 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 13158
I20260621 02:44:33.796497 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 13291
Jun 21 02:44:33 dist-test-slave-hvgn krb5kdc[12918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009873, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.166.3@KRBTEST.COM
I20260621 02:44:33.812191 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:33.797593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54704 (local address 127.9.166.3:40267)
0621 02:44:33.798183 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:33.798190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:33.798217 (+ 27us) server_negotiation.cc:407] Connection header received
0621 02:44:33.798275 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:33.798281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:33.798477 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:33.798652 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:33.800024 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:33.801058 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:33.802272 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:33.802821 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:33.802958 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:33.807145 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:33.807156 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:33.807170 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:33.807205 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:33.809827 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:33.810422 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:33.810428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:33.810430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:33.810534 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:33.810818 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:33.810821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:33.810823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:33.811122 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:33.811338 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:33.811625 (+ 287us) server_negotiation.cc:299] Negotiation successful
0621 02:44:33.811811 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":295,"threads_started":1}
W20260621 02:44:33.813459 13414 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54704
W20260621 02:44:33.814535 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 13291: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:44:33.814743 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 13291
I20260621 02:44:33.829932 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 12931
I20260621 02:44:33.831156 12993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:33.934391 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 12931
2026-06-21T02:44:33Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3508 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:44:34 dist-test-slave-hvgn krb5kdc[13436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:44:34 dist-test-slave-hvgn krb5kdc[13436](info): set up 2 sockets
Jun 21 02:44:34 dist-test-slave-hvgn krb5kdc[13436](info): commencing operation
krb5kdc: starting...
W20260621 02:44:35.011741 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:44:35 dist-test-slave-hvgn krb5kdc[13436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:44:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:44:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:44:35.192834 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:34141
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:36507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:34141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:35.513000 13449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:35.513316 13449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:35.513484 13449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:35.520403 13449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:44:35.520537 13449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:35.520622 13449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:35.520715 13449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:44:35.520789 13449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:44:35.530030 13449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:36507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:34141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:34141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:35.531275 13449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:35.532509 13449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:35.543876 13454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:35.543789 13457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:35.543820 13455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:35.544953 13449 server_base.cc:1034] running on GCE node
I20260621 02:44:35.545941 13449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:35.547355 13449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:35.548642 13449 hybrid_clock.cc:648] HybridClock initialized: now 1782009875548595 us; error 33 us; skew 500 ppm
I20260621 02:44:35.549006 13449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:35 dist-test-slave-hvgn krb5kdc[13436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009875, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:35.557050 13449 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:44:35.558512 13449 webserver.cc:460] Webserver started at http://127.9.166.62:37805/ using document root <none> and password file <none>
I20260621 02:44:35.559171 13449 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:35.559333 13449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:35.559671 13449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:35.563063 13449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "e77be4dc7bb341f880d7a7d22c114729"
format_stamp: "Formatted at 2026-06-21 02:44:35 on dist-test-slave-hvgn"
server_key: "a2b201fcea091cf409fab6b34c278b05"
server_key_iv: "2219d47035870cb3e1528472d8e897ce"
server_key_version: "encryptionkey@0"
I20260621 02:44:35.563874 13449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "e77be4dc7bb341f880d7a7d22c114729"
format_stamp: "Formatted at 2026-06-21 02:44:35 on dist-test-slave-hvgn"
server_key: "a2b201fcea091cf409fab6b34c278b05"
server_key_iv: "2219d47035870cb3e1528472d8e897ce"
server_key_version: "encryptionkey@0"
I20260621 02:44:35.568773 13449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:44:35.571960 13464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:35.573406 13449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:44:35.573608 13449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "e77be4dc7bb341f880d7a7d22c114729"
format_stamp: "Formatted at 2026-06-21 02:44:35 on dist-test-slave-hvgn"
server_key: "a2b201fcea091cf409fab6b34c278b05"
server_key_iv: "2219d47035870cb3e1528472d8e897ce"
server_key_version: "encryptionkey@0"
I20260621 02:44:35.573808 13449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:35.589022 13449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:35.592737 13449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:35.593004 13449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:35.611176 13449 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:34141
I20260621 02:44:35.611224 13515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:34141 every 8 connection(s)
I20260621 02:44:35.612488 13449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:44:35.616065 13516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:35.618913 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 13449
I20260621 02:44:35.619138 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:44:35.619956 9880 external_mini_cluster.cc:1428] Setting key 88982bd6c02336de23d09c99660da12f
I20260621 02:44:35.629017 13516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729: Bootstrap starting.
I20260621 02:44:35.632804 13516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:35.634074 13516 log.cc:824] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729: Log is configured to *not* fsync() on all Append() calls
Jun 21 02:44:35 dist-test-slave-hvgn krb5kdc[13436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:35.636763 13516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729: No bootstrap required, opened a new log
I20260621 02:44:35.643502 13519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:35.621963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56828 (local address 127.9.166.62:34141)
0621 02:44:35.622708 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:35.622718 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:35.622762 (+ 44us) server_negotiation.cc:407] Connection header received
0621 02:44:35.624436 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:35.624458 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:35.624836 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:35.625135 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:35.626941 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:35.629096 (+ 2155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:35.630881 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:35.631623 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:35.631747 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:35.636313 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:35.636321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:35.636334 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:35.636367 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:35.639658 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:35.640318 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:35.640325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:35.640330 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:35.640477 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:35.640861 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:35.640868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:35.640876 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:35.641381 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:35.641681 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:35.642093 (+ 412us) server_negotiation.cc:299] Negotiation successful
0621 02:44:35.642484 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":369,"threads_started":1}
I20260621 02:44:35.644634 13516 raft_consensus.cc:348] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77be4dc7bb341f880d7a7d22c114729" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34141 } }
I20260621 02:44:35.645067 13516 raft_consensus.cc:374] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:35.645188 13516 raft_consensus.cc:729] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77be4dc7bb341f880d7a7d22c114729, State: Initialized, Role: FOLLOWER
I20260621 02:44:35.645802 13516 consensus_queue.cc:260] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77be4dc7bb341f880d7a7d22c114729" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34141 } }
I20260621 02:44:35.646088 13516 raft_consensus.cc:388] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:44:35.646247 13516 raft_consensus.cc:482] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:44:35.646443 13516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:35.648607 13516 raft_consensus.cc:504] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77be4dc7bb341f880d7a7d22c114729" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34141 } }
I20260621 02:44:35.649168 13516 leader_election.cc:302] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e77be4dc7bb341f880d7a7d22c114729; no voters:
I20260621 02:44:35.649861 13516 leader_election.cc:288] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:44:35.649967 13521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:35.651528 13516 sys_catalog.cc:564] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:44:35.652061 13521 raft_consensus.cc:686] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729 [term 1 LEADER]: Becoming Leader. State: Replica: e77be4dc7bb341f880d7a7d22c114729, State: Running, Role: LEADER
I20260621 02:44:35.652542 13521 consensus_queue.cc:237] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77be4dc7bb341f880d7a7d22c114729" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34141 } }
I20260621 02:44:35.655457 13522 sys_catalog.cc:455] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e77be4dc7bb341f880d7a7d22c114729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77be4dc7bb341f880d7a7d22c114729" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34141 } } }
I20260621 02:44:35.655558 13523 sys_catalog.cc:455] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e77be4dc7bb341f880d7a7d22c114729. Latest consensus state: current_term: 1 leader_uuid: "e77be4dc7bb341f880d7a7d22c114729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77be4dc7bb341f880d7a7d22c114729" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34141 } } }
I20260621 02:44:35.655849 13522 sys_catalog.cc:458] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:35.655917 13523 sys_catalog.cc:458] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:35.660084 13531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:44:35.665129 13531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:44:35.672875 13531 catalog_manager.cc:1347] Generated new cluster ID: acc47d9978854ee19a6609e08e7f7c2b
I20260621 02:44:35.672995 13531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:44:35.685292 13531 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:44:35.686535 13531 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:44:35.694358 13531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e77be4dc7bb341f880d7a7d22c114729: Generated new TSK 0
I20260621 02:44:35.695196 13531 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
I20260621 02:44:35.745204 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34141
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:36507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:36.058562 13542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:36.058871 13542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:36.059056 13542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:36.066004 13542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:36.066141 13542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:36.066326 13542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:44:36.075438 13542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:36507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:36.076597 13542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:36.077939 13542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:36.090950 13547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:36.090982 13548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:36.091179 13550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:36.091935 13542 server_base.cc:1034] running on GCE node
I20260621 02:44:36.092484 13542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:36.093355 13542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:36.094583 13542 hybrid_clock.cc:648] HybridClock initialized: now 1782009876094549 us; error 56 us; skew 500 ppm
I20260621 02:44:36.094885 13542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:36 dist-test-slave-hvgn krb5kdc[13436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009876, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:36.098038 13542 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:44:36.099491 13542 webserver.cc:460] Webserver started at http://127.9.166.1:33491/ using document root <none> and password file <none>
I20260621 02:44:36.100134 13542 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:36.100281 13542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:36.100591 13542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:36.103329 13542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "cdecec6768394c009c2494a372f373bd"
format_stamp: "Formatted at 2026-06-21 02:44:36 on dist-test-slave-hvgn"
server_key: "6049af0c3913444ce6bed400b9fc0c66"
server_key_iv: "67c4739502358a42d73dc21bcb429b08"
server_key_version: "encryptionkey@0"
I20260621 02:44:36.104266 13542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "cdecec6768394c009c2494a372f373bd"
format_stamp: "Formatted at 2026-06-21 02:44:36 on dist-test-slave-hvgn"
server_key: "6049af0c3913444ce6bed400b9fc0c66"
server_key_iv: "67c4739502358a42d73dc21bcb429b08"
server_key_version: "encryptionkey@0"
I20260621 02:44:36.109624 13542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:44:36.113122 13557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:36.114406 13542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:44:36.114588 13542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "cdecec6768394c009c2494a372f373bd"
format_stamp: "Formatted at 2026-06-21 02:44:36 on dist-test-slave-hvgn"
server_key: "6049af0c3913444ce6bed400b9fc0c66"
server_key_iv: "67c4739502358a42d73dc21bcb429b08"
server_key_version: "encryptionkey@0"
I20260621 02:44:36.114782 13542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:36.126446 13542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:36.130481 13542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:36.130752 13542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:36.131685 13542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:36.132877 13542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:36.132977 13542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:36.133117 13542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:36.133188 13542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:36.165746 13542 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:42319
I20260621 02:44:36.165789 13669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:42319 every 8 connection(s)
I20260621 02:44:36.166937 13542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:44:36.173877 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 13542
I20260621 02:44:36.174127 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:44:36.174916 9880 external_mini_cluster.cc:1428] Setting key 4a63852613396e66cc94fe2a93d6264c
Jun 21 02:44:36 dist-test-slave-hvgn krb5kdc[13436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009876, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:36.187850 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:36.169414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:47735 (local address 127.9.166.62:34141)
0621 02:44:36.169818 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:36.169824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:36.170832 (+ 1008us) server_negotiation.cc:407] Connection header received
0621 02:44:36.172669 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:36.172674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:36.172764 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:36.172881 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:36.174786 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:36.175785 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:36.177863 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:36.178299 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:36.178392 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:36.182443 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:36.182447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:36.182458 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:36.182485 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:36.185044 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:36.185708 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:36.185715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:36.185718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:36.185831 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:36.186227 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:36.186235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:36.186239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:36.186495 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:36.186646 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:36.187251 (+ 605us) server_negotiation.cc:299] Negotiation successful
0621 02:44:36.187502 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":212,"threads_started":1}
I20260621 02:44:36.188732 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:36.170007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34141 (local address 127.9.166.1:47735)
0621 02:44:36.170675 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:36.170711 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:36.172412 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:36.173111 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:36.173119 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:36.173778 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:36.174606 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:36.174631 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:36.175945 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:36.175950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:36.177705 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:36.177731 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:36.178511 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:36.178521 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:36.178843 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:36.179642 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:36.179674 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:36.182235 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:36.185179 (+ 2944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:36.185189 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:36.185204 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:36.185594 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:36.185937 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:36.185943 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:36.185947 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:36.186122 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:36.186615 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:36.186623 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:36.186925 (+ 302us) client_negotiation.cc:769] Sending connection context
0621 02:44:36.187180 (+ 255us) client_negotiation.cc:240] Negotiation successful
0621 02:44:36.187680 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":265,"threads_started":1}
I20260621 02:44:36.190276 13670 heartbeater.cc:347] Connected to a master server at 127.9.166.62:34141
I20260621 02:44:36.190620 13670 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:36.191509 13670 heartbeater.cc:511] Master 127.9.166.62:34141 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.166.2@KRBTEST.COM; defaulting to no policy
I20260621 02:44:36.193917 13481 ts_manager.cc:194] Registered new tserver with Master: cdecec6768394c009c2494a372f373bd (127.9.166.1:42319)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.2@KRBTEST.COM" created.
I20260621 02:44:36.195525 13481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:47735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
I20260621 02:44:36.215898 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34141
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:36507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:36.532480 13676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:36.532804 13676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:36.533006 13676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:36.540304 13676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:36.540406 13676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:36.540619 13676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:44:36.550383 13676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:36507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:36.551745 13676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:36.553140 13676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:36.565670 13681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:36.565608 13682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:36.565680 13684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:36.566581 13676 server_base.cc:1034] running on GCE node
I20260621 02:44:36.567111 13676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:36.568046 13676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:36.569239 13676 hybrid_clock.cc:648] HybridClock initialized: now 1782009876569203 us; error 34 us; skew 500 ppm
I20260621 02:44:36.569563 13676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:36 dist-test-slave-hvgn krb5kdc[13436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009876, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:36.572580 13676 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:44:36.573928 13676 webserver.cc:460] Webserver started at http://127.9.166.2:46335/ using document root <none> and password file <none>
I20260621 02:44:36.574579 13676 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:36.574704 13676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:36.575008 13676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:36.577818 13676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "f79d32073a784e649be4ae97a6dd92b1"
format_stamp: "Formatted at 2026-06-21 02:44:36 on dist-test-slave-hvgn"
server_key: "0ca7409e83851ad85ba85f5cd4ae2010"
server_key_iv: "604ac267c3241475d972cd750d438ca6"
server_key_version: "encryptionkey@0"
I20260621 02:44:36.578564 13676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "f79d32073a784e649be4ae97a6dd92b1"
format_stamp: "Formatted at 2026-06-21 02:44:36 on dist-test-slave-hvgn"
server_key: "0ca7409e83851ad85ba85f5cd4ae2010"
server_key_iv: "604ac267c3241475d972cd750d438ca6"
server_key_version: "encryptionkey@0"
I20260621 02:44:36.583529 13676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 02:44:36.586670 13691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:36.587895 13676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 02:44:36.588069 13676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "f79d32073a784e649be4ae97a6dd92b1"
format_stamp: "Formatted at 2026-06-21 02:44:36 on dist-test-slave-hvgn"
server_key: "0ca7409e83851ad85ba85f5cd4ae2010"
server_key_iv: "604ac267c3241475d972cd750d438ca6"
server_key_version: "encryptionkey@0"
I20260621 02:44:36.588268 13676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:36.600160 13676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:36.603796 13676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:36.604074 13676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:36.604949 13676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:36.606150 13676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:36.606247 13676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:36.606392 13676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:36.606477 13676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:36.644943 13676 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:45139
I20260621 02:44:36.645031 13803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:45139 every 8 connection(s)
I20260621 02:44:36.646159 13676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:44:36.654803 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 13676
I20260621 02:44:36.655004 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:44:36.655560 9880 external_mini_cluster.cc:1428] Setting key 268d6ab4a9af30f271827576fe840a3a
Jun 21 02:44:36 dist-test-slave-hvgn krb5kdc[13436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009876, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:36.667076 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:36.648609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:34709 (local address 127.9.166.62:34141)
0621 02:44:36.648817 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:36.648823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:36.650023 (+ 1200us) server_negotiation.cc:407] Connection header received
0621 02:44:36.651782 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:36.651786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:36.651852 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:36.651936 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:36.653381 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:36.654161 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:36.656450 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:36.657015 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:36.657133 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:36.661071 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:36.661075 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:36.661078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:36.661115 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:36.664246 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:36.664922 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:36.664928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:36.664931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:36.665071 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:36.665502 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:36.665506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:36.665508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:36.665734 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:36.665899 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:36.666557 (+ 658us) server_negotiation.cc:299] Negotiation successful
0621 02:44:36.666765 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:44:36.668076 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:36.649144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34141 (local address 127.9.166.2:34709)
0621 02:44:36.649866 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:36.649902 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:36.651547 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:36.652123 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:36.652130 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:36.652728 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:36.653245 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:36.653260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:36.654313 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:36.654317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:36.656320 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:36.656333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:36.657230 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:36.657238 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:36.657541 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:36.658212 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:36.658242 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:36.660840 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:36.664383 (+ 3543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:36.664390 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:36.664404 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:36.664798 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:36.665180 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:36.665185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:36.665188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:36.665371 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:36.665857 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:36.665864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:36.666200 (+ 336us) client_negotiation.cc:769] Sending connection context
0621 02:44:36.666440 (+ 240us) client_negotiation.cc:240] Negotiation successful
0621 02:44:36.666911 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":318,"threads_started":1}
I20260621 02:44:36.669421 13804 heartbeater.cc:347] Connected to a master server at 127.9.166.62:34141
I20260621 02:44:36.669734 13804 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:36.670591 13804 heartbeater.cc:511] Master 127.9.166.62:34141 requested a full tablet report, sending...
I20260621 02:44:36.672219 13481 ts_manager.cc:194] Registered new tserver with Master: f79d32073a784e649be4ae97a6dd92b1 (127.9.166.2:45139)
I20260621 02:44:36.673421 13481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:34709
WARNING: no policy specified for kudu/127.9.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
I20260621 02:44:36.696508 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34141
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:36507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:37.008800 13809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:37.009060 13809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:37.009230 13809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:37.016215 13809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:37.016335 13809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:37.016513 13809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:44:37.026087 13809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:36507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:37.027316 13809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:37.028602 13809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:37.040599 13814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:37.040723 13817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:37.040759 13815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:37.041913 13809 server_base.cc:1034] running on GCE node
I20260621 02:44:37.042469 13809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:37.043524 13809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:37.044749 13809 hybrid_clock.cc:648] HybridClock initialized: now 1782009877044706 us; error 54 us; skew 500 ppm
I20260621 02:44:37.045060 13809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:37 dist-test-slave-hvgn krb5kdc[13436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009877, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:37.048102 13809 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:44:37.049472 13809 webserver.cc:460] Webserver started at http://127.9.166.3:33819/ using document root <none> and password file <none>
I20260621 02:44:37.050082 13809 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:37.050177 13809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:37.050421 13809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:37.053342 13809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "585c29cde686497c9643b855bcfb2bde"
format_stamp: "Formatted at 2026-06-21 02:44:37 on dist-test-slave-hvgn"
server_key: "0f18ae089d571a458dd39dbc28fdb256"
server_key_iv: "b3eab320406db8eb2b6946ddab652e2c"
server_key_version: "encryptionkey@0"
I20260621 02:44:37.054044 13809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "585c29cde686497c9643b855bcfb2bde"
format_stamp: "Formatted at 2026-06-21 02:44:37 on dist-test-slave-hvgn"
server_key: "0f18ae089d571a458dd39dbc28fdb256"
server_key_iv: "b3eab320406db8eb2b6946ddab652e2c"
server_key_version: "encryptionkey@0"
I20260621 02:44:37.059185 13809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:44:37.062422 13824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:37.063789 13809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:44:37.063956 13809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "585c29cde686497c9643b855bcfb2bde"
format_stamp: "Formatted at 2026-06-21 02:44:37 on dist-test-slave-hvgn"
server_key: "0f18ae089d571a458dd39dbc28fdb256"
server_key_iv: "b3eab320406db8eb2b6946ddab652e2c"
server_key_version: "encryptionkey@0"
I20260621 02:44:37.064145 13809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:37.076944 13809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:37.080730 13809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:37.081005 13809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:37.081933 13809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:37.083170 13809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:37.083240 13809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:37.083425 13809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:37.083491 13809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260621 02:44:37.116947 13809 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:37561
I20260621 02:44:37.117022 13936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:37561 every 8 connection(s)
I20260621 02:44:37.118268 13809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:44:37.122640 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 13809
I20260621 02:44:37.122823 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:44:37.123245 9880 external_mini_cluster.cc:1428] Setting key 25328422b77d306fa7f9b79602d7987c
Jun 21 02:44:37 dist-test-slave-hvgn krb5kdc[13436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009877, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:37.137315 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:37.120876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:43909 (local address 127.9.166.62:34141)
0621 02:44:37.121055 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:37.121060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:37.122212 (+ 1152us) server_negotiation.cc:407] Connection header received
0621 02:44:37.124400 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:37.124407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:37.124489 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:37.124616 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:37.126451 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:37.127173 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:37.128697 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:37.129093 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:37.129184 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:37.132338 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:37.132343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:37.132346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:37.132374 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:37.134690 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:37.135292 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:37.135295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:37.135297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:37.135453 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:37.135838 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:37.135841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:37.135843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:37.136083 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:37.136186 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:37.136815 (+ 629us) server_negotiation.cc:299] Negotiation successful
0621 02:44:37.137025 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:44:37.138432 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:37.121321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34141 (local address 127.9.166.3:43909)
0621 02:44:37.122062 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:37.122102 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:37.124080 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:37.124798 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:37.124805 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:37.125516 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:37.126302 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:37.126321 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:37.127304 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:37.127307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:37.128549 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:37.128560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:37.129270 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:37.129274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:37.129460 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:37.130072 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:37.130095 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:37.132157 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:37.134809 (+ 2652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:37.134815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:37.134826 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:37.135193 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:37.135545 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:37.135548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:37.135550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:37.135742 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:37.136180 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:37.136185 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:37.136447 (+ 262us) client_negotiation.cc:769] Sending connection context
0621 02:44:37.136710 (+ 263us) client_negotiation.cc:240] Negotiation successful
0621 02:44:37.137157 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":324,"threads_started":1}
I20260621 02:44:37.139758 13937 heartbeater.cc:347] Connected to a master server at 127.9.166.62:34141
I20260621 02:44:37.140035 13937 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:37.140724 13937 heartbeater.cc:511] Master 127.9.166.62:34141 requested a full tablet report, sending...
I20260621 02:44:37.142076 13481 ts_manager.cc:194] Registered new tserver with Master: 585c29cde686497c9643b855bcfb2bde (127.9.166.3:37561)
I20260621 02:44:37.142839 13481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:43909
I20260621 02:44:37.150610 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 02:44:37 dist-test-slave-hvgn krb5kdc[13436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 02:44:37 dist-test-slave-hvgn krb5kdc[13436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:44:37.198319 13670 heartbeater.cc:503] Master 127.9.166.62:34141 was elected leader, sending a full tablet report...
I20260621 02:44:37.198658 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:37.183711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46054 (local address 127.9.166.1:42319)
0621 02:44:37.184380 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:37.184389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:37.184420 (+ 31us) server_negotiation.cc:407] Connection header received
0621 02:44:37.184495 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:37.184501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:37.184704 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:37.184864 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:37.186172 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:37.187154 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:37.188401 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:37.188941 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:37.189075 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:37.192933 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:37.192944 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:37.192957 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:37.192987 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:37.195859 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:37.196483 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:37.196499 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:37.196501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:37.196625 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:37.197011 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:37.197016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:37.197021 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:37.197334 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:37.197639 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:37.197879 (+ 240us) server_negotiation.cc:299] Negotiation successful
0621 02:44:37.198144 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":306,"threads_started":1}
Jun 21 02:44:37 dist-test-slave-hvgn krb5kdc[13436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:37.215674 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:37.202789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56844 (local address 127.9.166.62:34141)
0621 02:44:37.202997 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:37.203004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:37.203244 (+ 240us) server_negotiation.cc:407] Connection header received
0621 02:44:37.203445 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:37.203450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:37.203520 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:37.203624 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:37.204807 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:37.205515 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:37.206816 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:37.207232 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:37.207360 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:37.210848 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:37.210852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:37.210855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:37.210885 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:37.213197 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:37.213790 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:37.213797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:37.213805 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:37.213920 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:37.214279 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:37.214285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:37.214288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:37.214592 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:37.214833 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:37.215137 (+ 304us) server_negotiation.cc:299] Negotiation successful
0621 02:44:37.215344 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260621 02:44:37.216199 13481 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56844
I20260621 02:44:37.217167 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 13542
I20260621 02:44:37.229234 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:37.218123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.9.166.1:42319)
0621 02:44:37.218304 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:37.218310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:37.218470 (+ 160us) server_negotiation.cc:407] Connection header received
0621 02:44:37.218645 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:37.218650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:37.218715 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:37.218810 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:37.220055 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:37.220709 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:37.221849 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:37.222269 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:37.222402 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:37.224572 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:37.224577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:37.224581 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:37.224609 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:37.226898 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:37.227496 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:37.227503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:37.227506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:37.227634 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:37.227987 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:37.227992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:37.227996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:37.228217 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:37.228450 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:37.228746 (+ 296us) server_negotiation.cc:299] Negotiation successful
0621 02:44:37.228952 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260621 02:44:37.229852 13665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:37.320231 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 13542
I20260621 02:44:37.336748 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 13676
Jun 21 02:44:37 dist-test-slave-hvgn krb5kdc[13436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.2@KRBTEST.COM
I20260621 02:44:37.352933 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:37.337719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.9.166.2:45139)
0621 02:44:37.338224 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:37.338231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:37.338255 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:44:37.338312 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:37.338319 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:37.338560 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:37.338692 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:37.339938 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:37.340935 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:37.342321 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:37.342853 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:37.342945 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:37.347070 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:37.347080 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:37.347094 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:37.347125 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:37.350136 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:37.350819 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:37.350826 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:37.350828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:37.350947 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:37.351301 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:37.351305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:37.351307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:37.351734 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:37.352036 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:37.352356 (+ 320us) server_negotiation.cc:299] Negotiation successful
0621 02:44:37.352580 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":255,"threads_started":1}
I20260621 02:44:37.354267 13799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:37.435783 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 13676
I20260621 02:44:37.451298 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 13809
Jun 21 02:44:37 dist-test-slave-hvgn krb5kdc[13436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.3@KRBTEST.COM
I20260621 02:44:37.468539 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:37.452475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45314 (local address 127.9.166.3:37561)
0621 02:44:37.453119 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:37.453127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:37.453155 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:44:37.453216 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:37.453223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:37.453434 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:37.453567 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:37.455015 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:37.456174 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:37.457526 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:37.458065 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:37.458168 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:37.462550 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:37.462560 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:37.462574 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:37.462606 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:37.465564 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:37.466176 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:37.466183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:37.466186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:37.466298 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:37.466742 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:37.466749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:37.466752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:37.467187 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:37.467542 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:37.467852 (+ 310us) server_negotiation.cc:299] Negotiation successful
0621 02:44:37.468132 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":326,"threads_started":1}
I20260621 02:44:37.469762 13932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:37.573486 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 13809
I20260621 02:44:37.589027 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 13449
I20260621 02:44:37.590242 13511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:37.693046 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 13449
2026-06-21T02:44:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3758 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:44:37 dist-test-slave-hvgn krb5kdc[13956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:44:37 dist-test-slave-hvgn krb5kdc[13956](info): set up 2 sockets
Jun 21 02:44:37 dist-test-slave-hvgn krb5kdc[13956](info): commencing operation
krb5kdc: starting...
W20260621 02:44:38.771952 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:44:38 dist-test-slave-hvgn krb5kdc[13956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:44:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:44:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:44:38.954702 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:37439
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:37439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:39.261749 13969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:39.262024 13969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:39.262115 13969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:39.271785 13969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:44:39.271891 13969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:39.271936 13969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:39.271993 13969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:44:39.272039 13969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:44:39.281664 13969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:37439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:37439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:39.282842 13969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:39.284126 13969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:39.295569 13974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:39.295478 13977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:39.295524 13975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:39.296372 13969 server_base.cc:1034] running on GCE node
I20260621 02:44:39.297147 13969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:39.298511 13969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:39.299736 13969 hybrid_clock.cc:648] HybridClock initialized: now 1782009879299703 us; error 55 us; skew 500 ppm
I20260621 02:44:39.300081 13969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:39 dist-test-slave-hvgn krb5kdc[13956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009879, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:39.303107 13969 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:44:39.304301 13969 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:39.304394 13969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:39.304643 13969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:39.307430 13969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "b3f672e1604746878ff5e8b1af298fd1"
format_stamp: "Formatted at 2026-06-21 02:44:39 on dist-test-slave-hvgn"
server_key: "9bdde2f66861516c02a92880d4cd1249"
server_key_iv: "f22d8cbecf0fa103afaed01800975419"
server_key_version: "encryptionkey@0"
I20260621 02:44:39.308169 13969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "b3f672e1604746878ff5e8b1af298fd1"
format_stamp: "Formatted at 2026-06-21 02:44:39 on dist-test-slave-hvgn"
server_key: "9bdde2f66861516c02a92880d4cd1249"
server_key_iv: "f22d8cbecf0fa103afaed01800975419"
server_key_version: "encryptionkey@0"
I20260621 02:44:39.312877 13969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260621 02:44:39.316092 13983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:39.317323 13969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:44:39.317507 13969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "b3f672e1604746878ff5e8b1af298fd1"
format_stamp: "Formatted at 2026-06-21 02:44:39 on dist-test-slave-hvgn"
server_key: "9bdde2f66861516c02a92880d4cd1249"
server_key_iv: "f22d8cbecf0fa103afaed01800975419"
server_key_version: "encryptionkey@0"
I20260621 02:44:39.317684 13969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:39.337409 13969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:39.341627 13969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:39.341900 13969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:39.359977 13969 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:37439
I20260621 02:44:39.360021 14034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:37439 every 8 connection(s)
I20260621 02:44:39.360894 13969 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:44:39.361498 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 13969
I20260621 02:44:39.361766 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:44:39.362485 9880 external_mini_cluster.cc:1428] Setting key b1f7c8dc424b7b46288302aafee73863
I20260621 02:44:39.365340 14035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:44:39 dist-test-slave-hvgn krb5kdc[13956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:39.377418 14035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1: Bootstrap starting.
I20260621 02:44:39.380591 14035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:39.381661 14035 log.cc:824] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:39.384371 14035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1: No bootstrap required, opened a new log
I20260621 02:44:39.385203 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:39.364422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40456 (local address 127.9.166.62:37439)
0621 02:44:39.365252 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:39.365269 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:39.365322 (+ 53us) server_negotiation.cc:407] Connection header received
0621 02:44:39.367635 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:39.367660 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:39.368143 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:39.368582 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:39.369899 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:39.371960 (+ 2061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:39.373141 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:39.374126 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:39.374296 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:39.378281 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:39.378289 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:39.378302 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:39.378338 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:39.381330 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:39.382082 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:39.382088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:39.382093 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:39.382255 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:39.382670 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:39.382675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:39.382685 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:39.383171 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:39.383472 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:39.383902 (+ 430us) server_negotiation.cc:299] Negotiation successful
0621 02:44:39.384269 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":387,"threads_started":1}
I20260621 02:44:39.392493 14035 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f672e1604746878ff5e8b1af298fd1" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37439 } }
I20260621 02:44:39.392819 14035 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:39.392936 14035 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f672e1604746878ff5e8b1af298fd1, State: Initialized, Role: FOLLOWER
I20260621 02:44:39.393510 14035 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f672e1604746878ff5e8b1af298fd1" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37439 } }
I20260621 02:44:39.393738 14035 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:44:39.393862 14035 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:44:39.394037 14035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:39.396101 14035 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f672e1604746878ff5e8b1af298fd1" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37439 } }
I20260621 02:44:39.396575 14035 leader_election.cc:302] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3f672e1604746878ff5e8b1af298fd1; no voters:
I20260621 02:44:39.397153 14035 leader_election.cc:288] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:44:39.397253 14040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:39.399063 14040 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1 [term 1 LEADER]: Becoming Leader. State: Replica: b3f672e1604746878ff5e8b1af298fd1, State: Running, Role: LEADER
I20260621 02:44:39.399569 14040 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f672e1604746878ff5e8b1af298fd1" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37439 } }
I20260621 02:44:39.399937 14035 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:44:39.402383 14042 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3f672e1604746878ff5e8b1af298fd1. Latest consensus state: current_term: 1 leader_uuid: "b3f672e1604746878ff5e8b1af298fd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f672e1604746878ff5e8b1af298fd1" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37439 } } }
I20260621 02:44:39.402760 14042 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:39.403312 14041 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3f672e1604746878ff5e8b1af298fd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3f672e1604746878ff5e8b1af298fd1" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37439 } } }
I20260621 02:44:39.403621 14041 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:39.404767 14049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:44:39.409246 14049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:44:39.417420 14049 catalog_manager.cc:1347] Generated new cluster ID: 2f94259df3c04c778518d80a14abcce8
I20260621 02:44:39.417514 14049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:44:39.436479 14049 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:44:39.437678 14049 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:44:39.443629 14049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1: Generated new TSK 0
I20260621 02:44:39.444566 14049 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
I20260621 02:44:39.488602 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:37439
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:39.812057 14061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:39.812320 14061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:39.812505 14061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:39.819792 14061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:39.819912 14061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:39.820080 14061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:44:39.830408 14061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:37439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:39.831761 14061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:39.833086 14061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:39.846097 14069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:39.846065 14066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:39.846065 14067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:39.846911 14061 server_base.cc:1034] running on GCE node
I20260621 02:44:39.847609 14061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:39.848547 14061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:39.849805 14061 hybrid_clock.cc:648] HybridClock initialized: now 1782009879849739 us; error 49 us; skew 500 ppm
I20260621 02:44:39.850147 14061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:39 dist-test-slave-hvgn krb5kdc[13956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009879, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:39.853330 14061 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:44:39.854481 14061 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:39.854614 14061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:39.854923 14061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:39.857824 14061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "198564ec0c424217a680223e316298c3"
format_stamp: "Formatted at 2026-06-21 02:44:39 on dist-test-slave-hvgn"
server_key: "e50df2b1066eae212cf2b3c2b5bc7926"
server_key_iv: "8bef5a6e54b91bdafc92fa0e6c35ad9c"
server_key_version: "encryptionkey@0"
I20260621 02:44:39.858646 14061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "198564ec0c424217a680223e316298c3"
format_stamp: "Formatted at 2026-06-21 02:44:39 on dist-test-slave-hvgn"
server_key: "e50df2b1066eae212cf2b3c2b5bc7926"
server_key_iv: "8bef5a6e54b91bdafc92fa0e6c35ad9c"
server_key_version: "encryptionkey@0"
I20260621 02:44:39.863803 14061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260621 02:44:39.867195 14075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:39.868554 14061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:44:39.868734 14061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "198564ec0c424217a680223e316298c3"
format_stamp: "Formatted at 2026-06-21 02:44:39 on dist-test-slave-hvgn"
server_key: "e50df2b1066eae212cf2b3c2b5bc7926"
server_key_iv: "8bef5a6e54b91bdafc92fa0e6c35ad9c"
server_key_version: "encryptionkey@0"
I20260621 02:44:39.868952 14061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:39.888907 14061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:39.893335 14061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:39.893630 14061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:39.894647 14061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:39.896100 14061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:39.896224 14061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:39.896353 14061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:39.896440 14061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:39.929077 14061 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:43681
I20260621 02:44:39.929131 14187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:43681 every 8 connection(s)
I20260621 02:44:39.929932 14061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:44:39.935626 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 14061
I20260621 02:44:39.935860 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:44:39.936496 9880 external_mini_cluster.cc:1428] Setting key cf27d89b2c44840b06d899e89f96530c
Jun 21 02:44:39 dist-test-slave-hvgn krb5kdc[13956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009879, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:39.950690 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:39.932453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:55503 (local address 127.9.166.62:37439)
0621 02:44:39.932897 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:39.932906 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:39.933821 (+ 915us) server_negotiation.cc:407] Connection header received
0621 02:44:39.935765 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:39.935770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:39.935861 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:39.935981 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:39.937493 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:39.938211 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:39.939770 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:39.940167 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:39.940299 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:39.943678 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:39.943687 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:39.943695 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:39.943739 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:39.947381 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:39.948178 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:39.948186 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:39.948192 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:39.948334 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:39.948730 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:39.948737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:39.948743 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:39.949121 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:39.949267 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:39.949914 (+ 647us) server_negotiation.cc:299] Negotiation successful
0621 02:44:39.950175 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":209,"threads_started":1}
I20260621 02:44:39.951346 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:39.933011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37439 (local address 127.9.166.1:55503)
0621 02:44:39.933667 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:39.933709 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:39.935501 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:39.936215 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:39.936223 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:39.936787 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:39.937325 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:39.937341 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:39.938356 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:39.938366 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:39.939628 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:39.939639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:39.940381 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:39.940391 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:39.940613 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:39.941265 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:39.941286 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:39.943453 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:39.947565 (+ 4112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:39.947572 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:39.947583 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:39.948039 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:39.948439 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:39.948442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:39.948444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:39.948609 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:39.949253 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:39.949259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:39.949543 (+ 284us) client_negotiation.cc:769] Sending connection context
0621 02:44:39.949761 (+ 218us) client_negotiation.cc:240] Negotiation successful
0621 02:44:39.950178 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":270,"threads_started":1}
I20260621 02:44:39.953042 14188 heartbeater.cc:347] Connected to a master server at 127.9.166.62:37439
I20260621 02:44:39.953349 14188 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:39.954005 14188 heartbeater.cc:511] Master 127.9.166.62:37439 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.166.2@KRBTEST.COM; defaulting to no policy
I20260621 02:44:39.956171 14000 ts_manager.cc:194] Registered new tserver with Master: 198564ec0c424217a680223e316298c3 (127.9.166.1:43681)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.2@KRBTEST.COM" created.
I20260621 02:44:39.957664 14000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:55503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
I20260621 02:44:39.979292 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:37439
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:40.288472 14194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:40.288717 14194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:40.288807 14194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:40.295770 14194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:40.295863 14194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:40.295972 14194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:44:40.305645 14194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:37439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:40.306759 14194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:40.308025 14194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:40.320719 14199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:40.320731 14200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:40.320747 14202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:40.321733 14194 server_base.cc:1034] running on GCE node
I20260621 02:44:40.322359 14194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:40.323294 14194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:40.324501 14194 hybrid_clock.cc:648] HybridClock initialized: now 1782009880324439 us; error 47 us; skew 500 ppm
I20260621 02:44:40.324823 14194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:40 dist-test-slave-hvgn krb5kdc[13956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009880, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:40.327898 14194 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:44:40.329085 14194 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:40.329216 14194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:40.329519 14194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:40.332396 14194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "59e3f2b181cc4091bcdfca5528320c45"
format_stamp: "Formatted at 2026-06-21 02:44:40 on dist-test-slave-hvgn"
server_key: "159d2b9d0dfce18650c6e2aef3530e53"
server_key_iv: "8ffa5002abb8bcf13771bce74b69b3da"
server_key_version: "encryptionkey@0"
I20260621 02:44:40.333194 14194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "59e3f2b181cc4091bcdfca5528320c45"
format_stamp: "Formatted at 2026-06-21 02:44:40 on dist-test-slave-hvgn"
server_key: "159d2b9d0dfce18650c6e2aef3530e53"
server_key_iv: "8ffa5002abb8bcf13771bce74b69b3da"
server_key_version: "encryptionkey@0"
I20260621 02:44:40.337901 14194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:44:40.341251 14208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:40.342521 14194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:44:40.342689 14194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "59e3f2b181cc4091bcdfca5528320c45"
format_stamp: "Formatted at 2026-06-21 02:44:40 on dist-test-slave-hvgn"
server_key: "159d2b9d0dfce18650c6e2aef3530e53"
server_key_iv: "8ffa5002abb8bcf13771bce74b69b3da"
server_key_version: "encryptionkey@0"
I20260621 02:44:40.342896 14194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:40.360652 14194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:40.364655 14194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:40.364974 14194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:40.365937 14194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:40.367341 14194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:40.367478 14194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:40.367615 14194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:40.367707 14194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:40.399873 14194 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:44095
I20260621 02:44:40.399930 14320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:44095 every 8 connection(s)
I20260621 02:44:40.400781 14194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:44:40.405763 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 14194
I20260621 02:44:40.405972 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:44:40.406468 9880 external_mini_cluster.cc:1428] Setting key 3fb701b727d6cbac7aecc884d9792479
Jun 21 02:44:40 dist-test-slave-hvgn krb5kdc[13956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009880, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:40.421422 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:40.403091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:54137 (local address 127.9.166.62:37439)
0621 02:44:40.403258 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:40.403264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:40.404451 (+ 1187us) server_negotiation.cc:407] Connection header received
0621 02:44:40.406405 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:40.406412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:40.406503 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:40.406621 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:40.408469 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.409567 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.411041 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.411771 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.411906 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:40.415543 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:40.415548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:40.415551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:40.415580 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:40.418106 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:40.419000 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:40.419004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:40.419007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:40.419134 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:40.419628 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:40.419631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:40.419633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:40.419877 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:40.420020 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:40.420861 (+ 841us) server_negotiation.cc:299] Negotiation successful
0621 02:44:40.421045 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:44:40.422811 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:40.403601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37439 (local address 127.9.166.2:54137)
0621 02:44:40.404293 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:40.404328 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:40.406159 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:40.406817 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:40.406825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:40.407507 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:40.408302 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:40.408336 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.409718 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.409722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:40.410896 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:40.410907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.411955 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.411960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:40.412127 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:40.412789 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:40.412809 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:40.415329 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:40.418239 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:40.418250 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:40.418268 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:40.418861 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:40.419248 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:40.419255 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:40.419262 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:40.419512 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:40.420016 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:40.420027 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:40.420470 (+ 443us) client_negotiation.cc:769] Sending connection context
0621 02:44:40.420764 (+ 294us) client_negotiation.cc:240] Negotiation successful
0621 02:44:40.421374 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":318,"threads_started":1}
I20260621 02:44:40.424352 14321 heartbeater.cc:347] Connected to a master server at 127.9.166.62:37439
I20260621 02:44:40.424669 14321 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:40.425308 14321 heartbeater.cc:511] Master 127.9.166.62:37439 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.166.3@KRBTEST.COM; defaulting to no policy
I20260621 02:44:40.426733 14000 ts_manager.cc:194] Registered new tserver with Master: 59e3f2b181cc4091bcdfca5528320c45 (127.9.166.2:44095)
I20260621 02:44:40.427538 14000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:54137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
I20260621 02:44:40.450146 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:37439
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:40.761590 14326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:40.761842 14326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:40.761935 14326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:40.768797 14326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:40.768888 14326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:40.768994 14326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:44:40.778079 14326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:37439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:40.779166 14326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:40.780462 14326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:40.792573 14334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:40.792665 14331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:40.792639 14332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:40.793356 14326 server_base.cc:1034] running on GCE node
I20260621 02:44:40.793936 14326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:40.794780 14326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:40.795974 14326 hybrid_clock.cc:648] HybridClock initialized: now 1782009880795909 us; error 57 us; skew 500 ppm
I20260621 02:44:40.796317 14326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:40 dist-test-slave-hvgn krb5kdc[13956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009880, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:40.802984 14326 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:44:40.804330 14326 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:40.804466 14326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:40.804765 14326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:40.807715 14326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "1d3bd120fbe4439491be33acd791dbc4"
format_stamp: "Formatted at 2026-06-21 02:44:40 on dist-test-slave-hvgn"
server_key: "5a8eeafd69c631aeee8d712eeea5ce95"
server_key_iv: "07e5522cb0d07f42e743540bc7d1e0fe"
server_key_version: "encryptionkey@0"
I20260621 02:44:40.808499 14326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "1d3bd120fbe4439491be33acd791dbc4"
format_stamp: "Formatted at 2026-06-21 02:44:40 on dist-test-slave-hvgn"
server_key: "5a8eeafd69c631aeee8d712eeea5ce95"
server_key_iv: "07e5522cb0d07f42e743540bc7d1e0fe"
server_key_version: "encryptionkey@0"
I20260621 02:44:40.813295 14326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:44:40.816612 14340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:40.817868 14326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260621 02:44:40.818060 14326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "1d3bd120fbe4439491be33acd791dbc4"
format_stamp: "Formatted at 2026-06-21 02:44:40 on dist-test-slave-hvgn"
server_key: "5a8eeafd69c631aeee8d712eeea5ce95"
server_key_iv: "07e5522cb0d07f42e743540bc7d1e0fe"
server_key_version: "encryptionkey@0"
I20260621 02:44:40.818276 14326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:40.832789 14326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:40.836737 14326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:40.836989 14326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:40.837882 14326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:40.839141 14326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:40.839236 14326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:40.839359 14326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:40.839478 14326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260621 02:44:40.871186 14326 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:41479
I20260621 02:44:40.871234 14452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:41479 every 8 connection(s)
I20260621 02:44:40.872061 14326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:44:40.877228 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 14326
I20260621 02:44:40.877432 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:44:40.877851 9880 external_mini_cluster.cc:1428] Setting key 70a4c0d743ec1b84c4a75b04c48fe4bf
Jun 21 02:44:40 dist-test-slave-hvgn krb5kdc[13956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009880, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:40.891758 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:40.874601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:43047 (local address 127.9.166.62:37439)
0621 02:44:40.874786 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:40.874790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:40.875937 (+ 1147us) server_negotiation.cc:407] Connection header received
0621 02:44:40.877864 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:40.877871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:40.877990 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:40.878111 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:40.880056 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.880983 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.882444 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.882848 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.882939 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:40.886439 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:40.886444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:40.886447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:40.886479 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:40.888952 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:40.889655 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:40.889659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:40.889661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:40.889766 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:40.890129 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:40.890132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:40.890134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:40.890383 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:40.890494 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:40.891152 (+ 658us) server_negotiation.cc:299] Negotiation successful
0621 02:44:40.891316 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 02:44:40.892903 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:40.875079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37439 (local address 127.9.166.3:43047)
0621 02:44:40.875778 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:40.875821 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:40.877599 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:40.878363 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:40.878374 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:40.879103 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:40.879873 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:40.879905 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.881115 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.881119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:40.882326 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:40.882342 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.883095 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.883100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:40.883284 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:40.883951 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:40.883974 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:40.886204 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:40.889085 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:40.889092 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:40.889105 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:40.889547 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:40.889854 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:40.889857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:40.889859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:40.890023 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:40.890487 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:40.890493 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:40.890806 (+ 313us) client_negotiation.cc:769] Sending connection context
0621 02:44:40.891022 (+ 216us) client_negotiation.cc:240] Negotiation successful
0621 02:44:40.891588 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":305,"threads_started":1}
I20260621 02:44:40.894164 14453 heartbeater.cc:347] Connected to a master server at 127.9.166.62:37439
I20260621 02:44:40.894423 14453 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:40.895033 14453 heartbeater.cc:511] Master 127.9.166.62:37439 requested a full tablet report, sending...
I20260621 02:44:40.896420 14000 ts_manager.cc:194] Registered new tserver with Master: 1d3bd120fbe4439491be33acd791dbc4 (127.9.166.3:41479)
I20260621 02:44:40.897224 14000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:43047
I20260621 02:44:40.905146 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:44:40.921370 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:40.909580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40468 (local address 127.9.166.62:37439)
0621 02:44:40.909740 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:40.909745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:40.910258 (+ 513us) server_negotiation.cc:407] Connection header received
0621 02:44:40.910384 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:40.910388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:40.910442 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:40.910516 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:40.911978 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.912660 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.913844 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.914250 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.914334 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:40.916609 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:40.916613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:40.916616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:40.916645 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:40.918915 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:40.919538 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:40.919542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:40.919544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:40.919670 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:40.920000 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:40.920003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:40.920005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:40.920290 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:40.920453 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:40.920790 (+ 337us) server_negotiation.cc:299] Negotiation successful
0621 02:44:40.921018 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:44:40.925421 14000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:44:40.932286 14000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:44:40.954830 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:40.944235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40484 (local address 127.9.166.1:43681)
0621 02:44:40.946468 (+ 2233us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:40.946474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:40.946502 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:44:40.946578 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:40.946584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:40.946855 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:40.947101 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:40.947997 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.949067 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.952367 (+ 3300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.953709 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.953852 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:40.954090 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:40.954343 (+ 253us) server_negotiation.cc:299] Negotiation successful
0621 02:44:40.954502 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2092,"thread_start_us":1975,"threads_started":1}
I20260621 02:44:40.954918 14467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:40.944076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:43681 (local address 127.0.0.1:40484)
0621 02:44:40.945618 (+ 1542us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:40.945665 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:40.945880 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:40.947367 (+ 1487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:40.947374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:40.947430 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:40.947829 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:40.947842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.949194 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.949201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:40.952200 (+ 2999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:40.952215 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.953903 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.953909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:40.954116 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:40.954176 (+ 60us) client_negotiation.cc:769] Sending connection context
0621 02:44:40.954307 (+ 131us) client_negotiation.cc:240] Negotiation successful
0621 02:44:40.954532 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1215,"mutex_wait_us":24,"thread_start_us":5285,"threads_started":1}
I20260621 02:44:40.958352 14465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:40.944230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:41479 (local address 127.0.0.1:60874)
0621 02:44:40.949664 (+ 5434us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:40.949682 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:40.949805 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:40.950853 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:40.950858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:40.950881 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:40.951302 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:40.951313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.953357 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.953366 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:40.955701 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:40.955714 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.957574 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.957580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:40.957737 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:40.957767 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 02:44:40.957853 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 02:44:40.957974 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5299,"mutex_wait_us":4,"thread_start_us":1012,"threads_started":1}
I20260621 02:44:40.958426 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:40.944974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60874 (local address 127.9.166.3:41479)
0621 02:44:40.950316 (+ 5342us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:40.950324 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:40.950354 (+ 30us) server_negotiation.cc:407] Connection header received
0621 02:44:40.950434 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:40.950441 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:40.950660 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:40.951543 (+ 883us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:40.951794 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.953221 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.955879 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.957363 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.957502 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:40.957732 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:40.957925 (+ 193us) server_negotiation.cc:299] Negotiation successful
0621 02:44:40.958095 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5184,"thread_start_us":5034,"threads_started":1}
I20260621 02:44:40.958993 14123 tablet_service.cc:1467] Processing CreateTablet for tablet 109a6ef8b34744df911e5da34037c3e5 (DEFAULT_TABLE table=test-table [id=3f41819deaae4ddfa33b1887f7de3da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:40.959987 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:40.950766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:44095 (local address 127.0.0.1:56356)
0621 02:44:40.952256 (+ 1490us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:40.952269 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:40.952477 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:40.953059 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:40.953063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:40.953090 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:40.953495 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:40.953517 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.955334 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.955339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:40.957409 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:40.957424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.959238 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.959244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:40.959421 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:40.959453 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 02:44:40.959544 (+ 91us) client_negotiation.cc:240] Negotiation successful
0621 02:44:40.959670 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1394,"thread_start_us":1222,"threads_started":1}
I20260621 02:44:40.960124 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:40.950191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56356 (local address 127.9.166.2:44095)
0621 02:44:40.950721 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:40.950728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:40.952378 (+ 1650us) server_negotiation.cc:407] Connection header received
0621 02:44:40.952671 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:40.952679 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:40.952871 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:40.953053 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:40.953679 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.955171 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.957601 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:40.959045 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:40.959169 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:40.959331 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:40.959629 (+ 298us) server_negotiation.cc:299] Negotiation successful
0621 02:44:40.959791 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":307,"threads_started":1}
I20260621 02:44:40.960467 14188 heartbeater.cc:503] Master 127.9.166.62:37439 was elected leader, sending a full tablet report...
I20260621 02:44:40.961752 14123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 109a6ef8b34744df911e5da34037c3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:40.962256 14388 tablet_service.cc:1467] Processing CreateTablet for tablet 109a6ef8b34744df911e5da34037c3e5 (DEFAULT_TABLE table=test-table [id=3f41819deaae4ddfa33b1887f7de3da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:40.963586 14388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 109a6ef8b34744df911e5da34037c3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:40.963635 14256 tablet_service.cc:1467] Processing CreateTablet for tablet 109a6ef8b34744df911e5da34037c3e5 (DEFAULT_TABLE table=test-table [id=3f41819deaae4ddfa33b1887f7de3da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:40.964870 14256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 109a6ef8b34744df911e5da34037c3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:40.972661 14471 tablet_bootstrap.cc:492] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3: Bootstrap starting.
I20260621 02:44:40.976315 14471 tablet_bootstrap.cc:654] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:40.977484 14471 log.cc:824] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:40.977547 14472 tablet_bootstrap.cc:492] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4: Bootstrap starting.
I20260621 02:44:40.979364 14473 tablet_bootstrap.cc:492] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45: Bootstrap starting.
I20260621 02:44:40.980829 14471 tablet_bootstrap.cc:492] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3: No bootstrap required, opened a new log
I20260621 02:44:40.981098 14471 ts_tablet_manager.cc:1397] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260621 02:44:40.981293 14472 tablet_bootstrap.cc:654] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:40.982471 14472 log.cc:824] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:40.982839 14473 tablet_bootstrap.cc:654] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:40.983923 14473 log.cc:824] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:40.985328 14472 tablet_bootstrap.cc:492] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4: No bootstrap required, opened a new log
I20260621 02:44:40.985584 14472 ts_tablet_manager.cc:1397] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260621 02:44:40.986660 14473 tablet_bootstrap.cc:492] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45: No bootstrap required, opened a new log
I20260621 02:44:40.986953 14473 ts_tablet_manager.cc:1397] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260621 02:44:40.989667 14471 raft_consensus.cc:348] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198564ec0c424217a680223e316298c3" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43681 } } peers { permanent_uuid: "1d3bd120fbe4439491be33acd791dbc4" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41479 } } peers { permanent_uuid: "59e3f2b181cc4091bcdfca5528320c45" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44095 } }
I20260621 02:44:40.990028 14471 raft_consensus.cc:374] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:40.990177 14471 raft_consensus.cc:729] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 198564ec0c424217a680223e316298c3, State: Initialized, Role: FOLLOWER
I20260621 02:44:40.990702 14471 consensus_queue.cc:260] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198564ec0c424217a680223e316298c3" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43681 } } peers { permanent_uuid: "1d3bd120fbe4439491be33acd791dbc4" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41479 } } peers { permanent_uuid: "59e3f2b181cc4091bcdfca5528320c45" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44095 } }
I20260621 02:44:40.992323 14471 ts_tablet_manager.cc:1428] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3: Time spent starting tablet: real 0.011s user 0.000s sys 0.012s
I20260621 02:44:40.996193 14472 raft_consensus.cc:348] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198564ec0c424217a680223e316298c3" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43681 } } peers { permanent_uuid: "1d3bd120fbe4439491be33acd791dbc4" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41479 } } peers { permanent_uuid: "59e3f2b181cc4091bcdfca5528320c45" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44095 } }
I20260621 02:44:40.996620 14472 raft_consensus.cc:374] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:40.996716 14472 raft_consensus.cc:729] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3bd120fbe4439491be33acd791dbc4, State: Initialized, Role: FOLLOWER
I20260621 02:44:40.997323 14472 consensus_queue.cc:260] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198564ec0c424217a680223e316298c3" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43681 } } peers { permanent_uuid: "1d3bd120fbe4439491be33acd791dbc4" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41479 } } peers { permanent_uuid: "59e3f2b181cc4091bcdfca5528320c45" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44095 } }
I20260621 02:44:40.998399 14453 heartbeater.cc:503] Master 127.9.166.62:37439 was elected leader, sending a full tablet report...
I20260621 02:44:40.998988 14472 ts_tablet_manager.cc:1428] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260621 02:44:40.999568 14473 raft_consensus.cc:348] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198564ec0c424217a680223e316298c3" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43681 } } peers { permanent_uuid: "1d3bd120fbe4439491be33acd791dbc4" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41479 } } peers { permanent_uuid: "59e3f2b181cc4091bcdfca5528320c45" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44095 } }
I20260621 02:44:40.999997 14473 raft_consensus.cc:374] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:41.000094 14473 raft_consensus.cc:729] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e3f2b181cc4091bcdfca5528320c45, State: Initialized, Role: FOLLOWER
I20260621 02:44:41.000712 14473 consensus_queue.cc:260] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198564ec0c424217a680223e316298c3" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43681 } } peers { permanent_uuid: "1d3bd120fbe4439491be33acd791dbc4" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41479 } } peers { permanent_uuid: "59e3f2b181cc4091bcdfca5528320c45" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44095 } }
I20260621 02:44:41.001772 14321 heartbeater.cc:503] Master 127.9.166.62:37439 was elected leader, sending a full tablet report...
I20260621 02:44:41.002382 14473 ts_tablet_manager.cc:1428] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260621 02:44:41.113262 14479 raft_consensus.cc:482] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:44:41.113600 14479 raft_consensus.cc:504] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198564ec0c424217a680223e316298c3" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43681 } } peers { permanent_uuid: "1d3bd120fbe4439491be33acd791dbc4" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41479 } } peers { permanent_uuid: "59e3f2b181cc4091bcdfca5528320c45" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44095 } }
I20260621 02:44:41.115447 14479 leader_election.cc:288] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 198564ec0c424217a680223e316298c3 (127.9.166.1:43681), 1d3bd120fbe4439491be33acd791dbc4 (127.9.166.3:41479)
I20260621 02:44:41.122782 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:41.115989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:58001 (local address 127.9.166.3:41479)
0621 02:44:41.116648 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:41.116656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:41.117226 (+ 570us) server_negotiation.cc:407] Connection header received
0621 02:44:41.117404 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:41.117408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:41.117463 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:41.117581 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:41.118287 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:41.118998 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:41.120830 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:41.121762 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:41.121856 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:41.121963 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:41.122344 (+ 381us) server_negotiation.cc:299] Negotiation successful
0621 02:44:41.122480 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472}
I20260621 02:44:41.122987 14481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:41.116052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:41479 (local address 127.9.166.2:58001)
0621 02:44:41.117076 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:41.117122 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:41.117276 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:41.117612 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:41.117616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:41.117704 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:41.118121 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:41.118131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:41.119140 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:41.119143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:41.120670 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:41.120682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:41.121960 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:41.121968 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:41.122176 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:41.122210 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 02:44:41.122307 (+ 97us) client_negotiation.cc:240] Negotiation successful
0621 02:44:41.122414 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":21,"spinlock_wait_cycles":23936,"thread_start_us":703,"threads_started":1}
I20260621 02:44:41.123075 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:41.115956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:44137 (local address 127.9.166.1:43681)
0621 02:44:41.116389 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:41.116399 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:41.117133 (+ 734us) server_negotiation.cc:407] Connection header received
0621 02:44:41.117340 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:41.117349 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:41.117429 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:41.117496 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:41.118134 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:41.118907 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:41.120975 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:41.122067 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:41.122193 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:41.122292 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:41.122706 (+ 414us) server_negotiation.cc:299] Negotiation successful
0621 02:44:41.122835 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260621 02:44:41.123170 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:41.115636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:43681 (local address 127.9.166.2:44137)
0621 02:44:41.117005 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:41.117021 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:41.117163 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:41.117539 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:41.117547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:41.117574 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:41.117970 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:41.117983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:41.119079 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:41.119086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:41.120718 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:41.120731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:41.122265 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:41.122289 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:41.122484 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:41.122515 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 02:44:41.122648 (+ 133us) client_negotiation.cc:240] Negotiation successful
0621 02:44:41.122766 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1240,"mutex_wait_us":59,"thread_start_us":1028,"threads_started":1}
W20260621 02:44:41.123569 14454 tablet.cc:2369] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:44:41.124157 14408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109a6ef8b34744df911e5da34037c3e5" candidate_uuid: "59e3f2b181cc4091bcdfca5528320c45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d3bd120fbe4439491be33acd791dbc4" is_pre_election: true
I20260621 02:44:41.124766 14408 raft_consensus.cc:2454] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59e3f2b181cc4091bcdfca5528320c45 in term 0.
I20260621 02:44:41.125460 14209 leader_election.cc:302] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d3bd120fbe4439491be33acd791dbc4, 59e3f2b181cc4091bcdfca5528320c45; no voters:
I20260621 02:44:41.125516 14143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109a6ef8b34744df911e5da34037c3e5" candidate_uuid: "59e3f2b181cc4091bcdfca5528320c45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198564ec0c424217a680223e316298c3" is_pre_election: true
I20260621 02:44:41.125849 14479 raft_consensus.cc:2781] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:44:41.125993 14479 raft_consensus.cc:482] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:44:41.126103 14479 raft_consensus.cc:3037] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:41.126077 14143 raft_consensus.cc:2454] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59e3f2b181cc4091bcdfca5528320c45 in term 0.
I20260621 02:44:41.128432 14479 raft_consensus.cc:504] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198564ec0c424217a680223e316298c3" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43681 } } peers { permanent_uuid: "1d3bd120fbe4439491be33acd791dbc4" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41479 } } peers { permanent_uuid: "59e3f2b181cc4091bcdfca5528320c45" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44095 } }
I20260621 02:44:41.129097 14479 leader_election.cc:288] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [CANDIDATE]: Term 1 election: Requested vote from peers 198564ec0c424217a680223e316298c3 (127.9.166.1:43681), 1d3bd120fbe4439491be33acd791dbc4 (127.9.166.3:41479)
I20260621 02:44:41.129479 14143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109a6ef8b34744df911e5da34037c3e5" candidate_uuid: "59e3f2b181cc4091bcdfca5528320c45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198564ec0c424217a680223e316298c3"
I20260621 02:44:41.129498 14408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109a6ef8b34744df911e5da34037c3e5" candidate_uuid: "59e3f2b181cc4091bcdfca5528320c45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d3bd120fbe4439491be33acd791dbc4"
I20260621 02:44:41.129683 14143 raft_consensus.cc:3037] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:41.129678 14408 raft_consensus.cc:3037] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:41.132004 14408 raft_consensus.cc:2454] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59e3f2b181cc4091bcdfca5528320c45 in term 1.
I20260621 02:44:41.132114 14143 raft_consensus.cc:2454] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59e3f2b181cc4091bcdfca5528320c45 in term 1.
I20260621 02:44:41.132421 14209 leader_election.cc:302] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d3bd120fbe4439491be33acd791dbc4, 59e3f2b181cc4091bcdfca5528320c45; no voters:
I20260621 02:44:41.132704 14479 raft_consensus.cc:2781] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:41.133235 14479 raft_consensus.cc:686] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [term 1 LEADER]: Becoming Leader. State: Replica: 59e3f2b181cc4091bcdfca5528320c45, State: Running, Role: LEADER
I20260621 02:44:41.133791 14479 consensus_queue.cc:237] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198564ec0c424217a680223e316298c3" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43681 } } peers { permanent_uuid: "1d3bd120fbe4439491be33acd791dbc4" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41479 } } peers { permanent_uuid: "59e3f2b181cc4091bcdfca5528320c45" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44095 } }
I20260621 02:44:41.138585 13999 catalog_manager.cc:5495] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59e3f2b181cc4091bcdfca5528320c45 (127.9.166.2). New cstate: current_term: 1 leader_uuid: "59e3f2b181cc4091bcdfca5528320c45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "198564ec0c424217a680223e316298c3" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d3bd120fbe4439491be33acd791dbc4" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59e3f2b181cc4091bcdfca5528320c45" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44095 } health_report { overall_health: HEALTHY } } }
W20260621 02:44:41.152588 14322 tablet.cc:2369] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:44:41.182401 14189 tablet.cc:2369] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:44:41.191200 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:41.185654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56372 (local address 127.9.166.2:44095)
0621 02:44:41.185879 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:41.185883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:41.185897 (+ 14us) server_negotiation.cc:407] Connection header received
0621 02:44:41.185979 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:41.185982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:41.186064 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:41.186539 (+ 475us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:44:41.186906 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:41.187748 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:41.189094 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:41.189500 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:41.189631 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:41.190109 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:44:41.190642 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:44:41.190806 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:41.190855 (+ 49us) server_negotiation.cc:299] Negotiation successful
0621 02:44:41.190934 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:44:41.199366 14143 raft_consensus.cc:1261] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3 [term 1 FOLLOWER]: Refusing update from remote peer 59e3f2b181cc4091bcdfca5528320c45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:44:41.199366 14408 raft_consensus.cc:1261] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4 [term 1 FOLLOWER]: Refusing update from remote peer 59e3f2b181cc4091bcdfca5528320c45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:44:41.200423 14482 consensus_queue.cc:1035] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d3bd120fbe4439491be33acd791dbc4" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:41.200796 14479 consensus_queue.cc:1035] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "198564ec0c424217a680223e316298c3" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:41.234156 14000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:44:41.234503 14000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:44:41.238356 14000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3f672e1604746878ff5e8b1af298fd1: Sending DeleteTablet for 3 replicas of tablet 109a6ef8b34744df911e5da34037c3e5
I20260621 02:44:41.239533 14123 tablet_service.cc:1514] Processing DeleteTablet for tablet 109a6ef8b34744df911e5da34037c3e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:41 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:40484
I20260621 02:44:41.239542 14388 tablet_service.cc:1514] Processing DeleteTablet for tablet 109a6ef8b34744df911e5da34037c3e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:41 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:60874
I20260621 02:44:41.240741 14256 tablet_service.cc:1514] Processing DeleteTablet for tablet 109a6ef8b34744df911e5da34037c3e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:41 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:56356
I20260621 02:44:41.241931 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 14061
I20260621 02:44:41.249668 14495 tablet_replica.cc:330] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3: stopping tablet replica
I20260621 02:44:41.250326 14495 raft_consensus.cc:2229] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:44:41.251057 14495 raft_consensus.cc:2258] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:41.253813 14495 ts_tablet_manager.cc:1905] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:41.261018 14495 ts_tablet_manager.cc:1918] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:44:41.261632 14496 tablet_replica.cc:330] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4: stopping tablet replica
I20260621 02:44:41.262138 14496 raft_consensus.cc:2229] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:44:41.262570 14497 tablet_replica.cc:330] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45: stopping tablet replica
I20260621 02:44:41.262709 14496 raft_consensus.cc:2258] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:41.262972 14497 raft_consensus.cc:2229] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:44:41.263636 14497 raft_consensus.cc:2258] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:41.265960 14497 ts_tablet_manager.cc:1905] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 21 02:44:41 dist-test-slave-hvgn krb5kdc[13956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:44:41.273970 14497 ts_tablet_manager.cc:1918] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:44:41.274235 14497 log.cc:1196] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/109a6ef8b34744df911e5da34037c3e5
I20260621 02:44:41.274909 14497 ts_tablet_manager.cc:1939] T 109a6ef8b34744df911e5da34037c3e5 P 59e3f2b181cc4091bcdfca5528320c45: Deleting consensus metadata
I20260621 02:44:41.275276 14496 ts_tablet_manager.cc:1905] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:41.276865 13986 catalog_manager.cc:4841] TS 59e3f2b181cc4091bcdfca5528320c45 (127.9.166.2:44095): tablet 109a6ef8b34744df911e5da34037c3e5 (table test-table [id=3f41819deaae4ddfa33b1887f7de3da9]) successfully deleted
I20260621 02:44:41.277056 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:41.243546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.9.166.1:43681)
0621 02:44:41.243724 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:41.243732 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:41.249712 (+ 5980us) server_negotiation.cc:407] Connection header received
0621 02:44:41.249903 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:41.249908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:41.250017 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:41.250125 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:41.255955 (+ 5830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:41.257508 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:41.260188 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:41.261102 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:41.261226 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:41.269273 (+ 8047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:41.269292 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:41.269310 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:41.269357 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:41.273379 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:41.274393 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:41.274445 (+ 52us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:41.274452 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:41.274641 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:41.275103 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:41.275107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:41.275112 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:41.275596 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:41.275867 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:41.276428 (+ 561us) server_negotiation.cc:299] Negotiation successful
0621 02:44:41.276669 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:44:41.277456 14495 log.cc:1196] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/109a6ef8b34744df911e5da34037c3e5
I20260621 02:44:41.278191 14495 ts_tablet_manager.cc:1939] T 109a6ef8b34744df911e5da34037c3e5 P 198564ec0c424217a680223e316298c3: Deleting consensus metadata
I20260621 02:44:41.278854 14183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:41.282573 14496 ts_tablet_manager.cc:1918] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:44:41.282795 14496 log.cc:1196] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/109a6ef8b34744df911e5da34037c3e5
I20260621 02:44:41.283495 14496 ts_tablet_manager.cc:1939] T 109a6ef8b34744df911e5da34037c3e5 P 1d3bd120fbe4439491be33acd791dbc4: Deleting consensus metadata
I20260621 02:44:41.285571 13984 catalog_manager.cc:4841] TS 1d3bd120fbe4439491be33acd791dbc4 (127.9.166.3:41479): tablet 109a6ef8b34744df911e5da34037c3e5 (table test-table [id=3f41819deaae4ddfa33b1887f7de3da9]) successfully deleted
I20260621 02:44:41.377902 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 14061
I20260621 02:44:41.378103 13986 catalog_manager.cc:4841] TS 198564ec0c424217a680223e316298c3 (127.9.166.1:43681): tablet 109a6ef8b34744df911e5da34037c3e5 (table test-table [id=3f41819deaae4ddfa33b1887f7de3da9]) successfully deleted
I20260621 02:44:41.402647 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 14194
Jun 21 02:44:41 dist-test-slave-hvgn krb5kdc[13956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.2@KRBTEST.COM
I20260621 02:44:41.422765 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:41.403743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56374 (local address 127.9.166.2:44095)
0621 02:44:41.403940 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:41.403947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:41.404211 (+ 264us) server_negotiation.cc:407] Connection header received
0621 02:44:41.404412 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:41.404418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:41.404503 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:41.404614 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:41.406143 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:41.407452 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:41.409265 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:41.409956 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:41.410074 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:41.415180 (+ 5106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:41.415193 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:41.415211 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:41.415255 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:41.419585 (+ 4330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:41.420370 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:41.420380 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:41.420383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:41.420562 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:41.420945 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:41.420950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:41.420954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:41.421444 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:41.421810 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:41.422140 (+ 330us) server_negotiation.cc:299] Negotiation successful
0621 02:44:41.422361 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:44:41.423549 14316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:41.537794 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 14194
I20260621 02:44:41.556305 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 14326
Jun 21 02:44:41 dist-test-slave-hvgn krb5kdc[13956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.3@KRBTEST.COM
I20260621 02:44:41.571076 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:41.557082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60888 (local address 127.9.166.3:41479)
0621 02:44:41.557265 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:41.557269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:41.557288 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:44:41.557387 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:41.557391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:41.557451 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:41.557537 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:41.558763 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:41.559548 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:41.560713 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:41.561066 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:41.561158 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:41.566022 (+ 4864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:41.566035 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:41.566050 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:41.566081 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:41.568794 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:41.569377 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:41.569384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:41.569387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:41.569498 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:41.569814 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:41.569817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:41.569819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:41.570131 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:41.570347 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:41.570570 (+ 223us) server_negotiation.cc:299] Negotiation successful
0621 02:44:41.570727 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:44:41.571798 14448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:41.660284 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 14326
I20260621 02:44:41.676358 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 13969
I20260621 02:44:41.677441 14030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:41.787286 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 13969
2026-06-21T02:44:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4094 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-21T02:44:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:44:41Z Disabled control of system clock
I20260621 02:44:41.853983 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:41399
--webserver_interface=127.9.166.62
--webserver_port=0
--builtin_ntp_servers=127.9.166.20:34087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:41399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 02:44:42.168329 14506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:42.168603 14506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:42.168686 14506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:42.175609 14506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:44:42.175704 14506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:42.175740 14506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:42.175796 14506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:44:42.175837 14506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:44:42.184677 14506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:34087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:41399
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:41399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:42.185883 14506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:42.187610 14506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:42.199236 14514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:42.199273 14511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:42.199301 14512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:42.200461 14506 server_base.cc:1034] running on GCE node
I20260621 02:44:42.201308 14506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:42.202751 14506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:42.203953 14506 hybrid_clock.cc:648] HybridClock initialized: now 1782009882203911 us; error 53 us; skew 500 ppm
I20260621 02:44:42.204350 14506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:44:42.206619 14506 webserver.cc:460] Webserver started at http://127.9.166.62:34561/ using document root <none> and password file <none>
I20260621 02:44:42.207324 14506 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:42.207468 14506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:42.207803 14506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:42.210758 14506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "d22cf058abf9409b94c06ad1a38dd85f"
format_stamp: "Formatted at 2026-06-21 02:44:42 on dist-test-slave-hvgn"
server_key: "65b3c5a0cd64e26e078d43c0cb9d58f1"
server_key_iv: "f88cc08c89311a32d563ae50991f282c"
server_key_version: "encryptionkey@0"
I20260621 02:44:42.211611 14506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "d22cf058abf9409b94c06ad1a38dd85f"
format_stamp: "Formatted at 2026-06-21 02:44:42 on dist-test-slave-hvgn"
server_key: "65b3c5a0cd64e26e078d43c0cb9d58f1"
server_key_iv: "f88cc08c89311a32d563ae50991f282c"
server_key_version: "encryptionkey@0"
I20260621 02:44:42.216698 14506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260621 02:44:42.220201 14520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:42.221552 14506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260621 02:44:42.221743 14506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "d22cf058abf9409b94c06ad1a38dd85f"
format_stamp: "Formatted at 2026-06-21 02:44:42 on dist-test-slave-hvgn"
server_key: "65b3c5a0cd64e26e078d43c0cb9d58f1"
server_key_iv: "f88cc08c89311a32d563ae50991f282c"
server_key_version: "encryptionkey@0"
I20260621 02:44:42.221943 14506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:42.227625 14506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:42.228384 14506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:42.228675 14506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:42.246397 14506 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:41399
I20260621 02:44:42.246438 14571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:41399 every 8 connection(s)
I20260621 02:44:42.247799 14506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:44:42.251251 14572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:42.251893 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 14506
I20260621 02:44:42.252084 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:44:42.252910 9880 external_mini_cluster.cc:1428] Setting key 4f99ef8ae74ec8442da769eae1b772db
I20260621 02:44:42.261009 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:42.255017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44618 (local address 127.9.166.62:41399)
0621 02:44:42.256048 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:42.256062 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:42.256102 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:44:42.257625 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:42.257647 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:42.258051 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:42.258377 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:42.259487 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:42.259493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:42.259506 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:44:42.259520 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:42.259582 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:44:42.259597 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:42.259761 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:42.259879 (+ 118us) server_negotiation.cc:299] Negotiation successful
0621 02:44:42.260154 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":413,"threads_started":1}
I20260621 02:44:42.264164 14572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f: Bootstrap starting.
I20260621 02:44:42.267211 14572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:42.268208 14572 log.cc:824] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:42.271028 14572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f: No bootstrap required, opened a new log
I20260621 02:44:42.277969 14572 raft_consensus.cc:348] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22cf058abf9409b94c06ad1a38dd85f" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41399 } }
I20260621 02:44:42.278283 14572 raft_consensus.cc:374] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:42.278352 14572 raft_consensus.cc:729] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22cf058abf9409b94c06ad1a38dd85f, State: Initialized, Role: FOLLOWER
I20260621 02:44:42.278806 14572 consensus_queue.cc:260] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22cf058abf9409b94c06ad1a38dd85f" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41399 } }
I20260621 02:44:42.279001 14572 raft_consensus.cc:388] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:44:42.279093 14572 raft_consensus.cc:482] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:44:42.279210 14572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:42.281394 14572 raft_consensus.cc:504] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22cf058abf9409b94c06ad1a38dd85f" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41399 } }
I20260621 02:44:42.281833 14572 leader_election.cc:302] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d22cf058abf9409b94c06ad1a38dd85f; no voters:
I20260621 02:44:42.282435 14572 leader_election.cc:288] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:44:42.282579 14577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:42.283308 14577 raft_consensus.cc:686] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f [term 1 LEADER]: Becoming Leader. State: Replica: d22cf058abf9409b94c06ad1a38dd85f, State: Running, Role: LEADER
I20260621 02:44:42.283999 14577 consensus_queue.cc:237] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22cf058abf9409b94c06ad1a38dd85f" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41399 } }
I20260621 02:44:42.284914 14572 sys_catalog.cc:564] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:44:42.286778 14578 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d22cf058abf9409b94c06ad1a38dd85f. Latest consensus state: current_term: 1 leader_uuid: "d22cf058abf9409b94c06ad1a38dd85f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22cf058abf9409b94c06ad1a38dd85f" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41399 } } }
I20260621 02:44:42.286803 14579 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d22cf058abf9409b94c06ad1a38dd85f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22cf058abf9409b94c06ad1a38dd85f" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41399 } } }
I20260621 02:44:42.287570 14579 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:42.287232 14578 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:42.292734 14583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:44:42.297964 14583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:44:42.306293 14583 catalog_manager.cc:1347] Generated new cluster ID: dad6ecc7f4634dad9cf5d3c39f60fd3c
I20260621 02:44:42.306397 14583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:44:42.323067 14583 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:44:42.323576 14583 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:44:42.328970 14583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f: Generated new TSK 0
I20260621 02:44:42.329684 14583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:44:42.334244 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:41399
--builtin_ntp_servers=127.9.166.20:34087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260621 02:44:42.653551 14596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:42.653965 14596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:42.654124 14596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:42.662454 14596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:42.662551 14596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:42.662731 14596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:44:42.671824 14596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:34087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:41399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:42.673055 14596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:42.674774 14596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:42.687415 14602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:42.687520 14601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:42.687520 14604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:42.688221 14596 server_base.cc:1034] running on GCE node
I20260621 02:44:42.688757 14596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:42.689566 14596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:42.690798 14596 hybrid_clock.cc:648] HybridClock initialized: now 1782009882690750 us; error 57 us; skew 500 ppm
I20260621 02:44:42.691116 14596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:44:42.693666 14596 webserver.cc:460] Webserver started at http://127.9.166.1:37525/ using document root <none> and password file <none>
I20260621 02:44:42.694352 14596 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:42.694468 14596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:42.694811 14596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:42.697568 14596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "18af4b546b3b417fa86bb470c50222c5"
format_stamp: "Formatted at 2026-06-21 02:44:42 on dist-test-slave-hvgn"
server_key: "764a29b317bb63ed1e70b19f9975c9ab"
server_key_iv: "1cc35dde7da6581c6e2f7849aaeedd0b"
server_key_version: "encryptionkey@0"
I20260621 02:44:42.698326 14596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "18af4b546b3b417fa86bb470c50222c5"
format_stamp: "Formatted at 2026-06-21 02:44:42 on dist-test-slave-hvgn"
server_key: "764a29b317bb63ed1e70b19f9975c9ab"
server_key_iv: "1cc35dde7da6581c6e2f7849aaeedd0b"
server_key_version: "encryptionkey@0"
I20260621 02:44:42.703032 14596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260621 02:44:42.706740 14610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:42.708084 14596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 02:44:42.708266 14596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "18af4b546b3b417fa86bb470c50222c5"
format_stamp: "Formatted at 2026-06-21 02:44:42 on dist-test-slave-hvgn"
server_key: "764a29b317bb63ed1e70b19f9975c9ab"
server_key_iv: "1cc35dde7da6581c6e2f7849aaeedd0b"
server_key_version: "encryptionkey@0"
I20260621 02:44:42.708499 14596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:42.714015 14596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:42.714805 14596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:42.715073 14596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:42.716061 14596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:42.717402 14596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:42.717531 14596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:42.717660 14596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:42.717746 14596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:42.749943 14596 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:40939
I20260621 02:44:42.749994 14722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:40939 every 8 connection(s)
I20260621 02:44:42.751259 14596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:44:42.752862 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 14596
I20260621 02:44:42.753070 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:44:42.753886 9880 external_mini_cluster.cc:1428] Setting key 5c6003993d9149c7345a9bb5b35fe381
I20260621 02:44:42.757069 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:41399
--builtin_ntp_servers=127.9.166.20:34087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260621 02:44:42.760660 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:42.754786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:56639 (local address 127.9.166.62:41399)
0621 02:44:42.754974 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:42.754979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:42.756597 (+ 1618us) server_negotiation.cc:407] Connection header received
0621 02:44:42.758328 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:42.758332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:42.758397 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:42.758518 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:42.759828 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:42.759834 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:42.759839 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:44:42.759863 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:42.759894 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:44:42.759906 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:42.760027 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:42.760263 (+ 236us) server_negotiation.cc:299] Negotiation successful
0621 02:44:42.760330 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:44:42.761586 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:42.755447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:41399 (local address 127.9.166.1:56639)
0621 02:44:42.756456 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:42.756496 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:42.758070 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:42.758741 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:42.758749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:42.758840 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:42.759534 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:44:42.759536 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:42.759573 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:44:42.759655 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:42.760000 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:42.760009 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:42.760017 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 02:44:42.760313 (+ 296us) client_negotiation.cc:240] Negotiation successful
0621 02:44:42.760539 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":290,"threads_started":1}
I20260621 02:44:42.763103 14723 heartbeater.cc:347] Connected to a master server at 127.9.166.62:41399
I20260621 02:44:42.763458 14723 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:42.764020 14723 heartbeater.cc:511] Master 127.9.166.62:41399 requested a full tablet report, sending...
I20260621 02:44:42.765982 14537 ts_manager.cc:194] Registered new tserver with Master: 18af4b546b3b417fa86bb470c50222c5 (127.9.166.1:40939)
W20260621 02:44:43.072170 14726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:43.072469 14726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:43.072638 14726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:43.079654 14726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:43.079775 14726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:43.079952 14726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:44:43.088784 14726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:34087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:41399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:43.090025 14726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:43.091778 14726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:43.104491 14731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:43.104558 14734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:43.104657 14732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:43.105480 14726 server_base.cc:1034] running on GCE node
I20260621 02:44:43.106063 14726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:43.106985 14726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:43.108219 14726 hybrid_clock.cc:648] HybridClock initialized: now 1782009883108157 us; error 52 us; skew 500 ppm
I20260621 02:44:43.108572 14726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:44:43.110826 14726 webserver.cc:460] Webserver started at http://127.9.166.2:45581/ using document root <none> and password file <none>
I20260621 02:44:43.111617 14726 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:43.111706 14726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:43.112023 14726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:43.114885 14726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "ffd1db50076741e5b3f0e1e84f3e33f0"
format_stamp: "Formatted at 2026-06-21 02:44:43 on dist-test-slave-hvgn"
server_key: "0cfff8f1af85c13d436d317b840827bc"
server_key_iv: "3cf0982fad514d70b482fd2ccb3776ad"
server_key_version: "encryptionkey@0"
I20260621 02:44:43.115675 14726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "ffd1db50076741e5b3f0e1e84f3e33f0"
format_stamp: "Formatted at 2026-06-21 02:44:43 on dist-test-slave-hvgn"
server_key: "0cfff8f1af85c13d436d317b840827bc"
server_key_iv: "3cf0982fad514d70b482fd2ccb3776ad"
server_key_version: "encryptionkey@0"
I20260621 02:44:43.120638 14726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:44:43.123996 14740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:43.125473 14726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:44:43.125631 14726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "ffd1db50076741e5b3f0e1e84f3e33f0"
format_stamp: "Formatted at 2026-06-21 02:44:43 on dist-test-slave-hvgn"
server_key: "0cfff8f1af85c13d436d317b840827bc"
server_key_iv: "3cf0982fad514d70b482fd2ccb3776ad"
server_key_version: "encryptionkey@0"
I20260621 02:44:43.125852 14726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:43.131157 14726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:43.131944 14726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:43.132237 14726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:43.133271 14726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:43.134611 14726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:43.134745 14726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:43.134869 14726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:43.134960 14726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:43.166555 14726 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:42121
I20260621 02:44:43.166606 14852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:42121 every 8 connection(s)
I20260621 02:44:43.167950 14726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:44:43.175978 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:43.170563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:44479 (local address 127.9.166.62:41399)
0621 02:44:43.170750 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:43.170755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:43.172134 (+ 1379us) server_negotiation.cc:407] Connection header received
0621 02:44:43.173797 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:43.173801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:43.173851 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:43.173926 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:43.175247 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:43.175251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:43.175255 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:44:43.175270 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:43.175301 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:44:43.175312 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:43.175465 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:43.175719 (+ 254us) server_negotiation.cc:299] Negotiation successful
0621 02:44:43.175765 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:44:43.176095 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 14726
I20260621 02:44:43.176261 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:44:43.176878 9880 external_mini_cluster.cc:1428] Setting key 26d5d2db85afeb1769471b51ae220d96
I20260621 02:44:43.177457 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:43.171058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:41399 (local address 127.9.166.2:44479)
0621 02:44:43.171985 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:43.172020 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:43.173547 (+ 1527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:43.174176 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:43.174183 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:43.174273 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:43.174948 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:44:43.174951 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:43.174990 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:44:43.175079 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:43.175470 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:43.175479 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:43.175486 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 02:44:43.175809 (+ 323us) client_negotiation.cc:240] Negotiation successful
0621 02:44:43.176096 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":317,"threads_started":1}
I20260621 02:44:43.178983 14853 heartbeater.cc:347] Connected to a master server at 127.9.166.62:41399
I20260621 02:44:43.179324 14853 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:43.180107 14853 heartbeater.cc:511] Master 127.9.166.62:41399 requested a full tablet report, sending...
I20260621 02:44:43.180143 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:41399
--builtin_ntp_servers=127.9.166.20:34087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260621 02:44:43.181700 14537 ts_manager.cc:194] Registered new tserver with Master: ffd1db50076741e5b3f0e1e84f3e33f0 (127.9.166.2:42121)
W20260621 02:44:43.501650 14856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:43.501959 14856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:43.502123 14856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:43.509311 14856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:43.509446 14856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:43.509627 14856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:44:43.518797 14856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:34087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:41399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:43.520129 14856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:43.521935 14856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:43.534876 14861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:43.534833 14864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:43.534806 14862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:43.536036 14856 server_base.cc:1034] running on GCE node
I20260621 02:44:43.536630 14856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:43.537535 14856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:43.538751 14856 hybrid_clock.cc:648] HybridClock initialized: now 1782009883538699 us; error 75 us; skew 500 ppm
I20260621 02:44:43.539057 14856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:44:43.541388 14856 webserver.cc:460] Webserver started at http://127.9.166.3:43893/ using document root <none> and password file <none>
I20260621 02:44:43.542085 14856 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:43.542213 14856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:43.542521 14856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:43.545280 14856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "19f720c0d7b143ce88e862d856b79559"
format_stamp: "Formatted at 2026-06-21 02:44:43 on dist-test-slave-hvgn"
server_key: "b117d4b421751f9b69d589dcc475d29d"
server_key_iv: "869326a4a6a9de1de9bd7d353d7f1685"
server_key_version: "encryptionkey@0"
I20260621 02:44:43.546051 14856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "19f720c0d7b143ce88e862d856b79559"
format_stamp: "Formatted at 2026-06-21 02:44:43 on dist-test-slave-hvgn"
server_key: "b117d4b421751f9b69d589dcc475d29d"
server_key_iv: "869326a4a6a9de1de9bd7d353d7f1685"
server_key_version: "encryptionkey@0"
I20260621 02:44:43.550920 14856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 02:44:43.554347 14870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:43.555696 14856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:44:43.555860 14856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "19f720c0d7b143ce88e862d856b79559"
format_stamp: "Formatted at 2026-06-21 02:44:43 on dist-test-slave-hvgn"
server_key: "b117d4b421751f9b69d589dcc475d29d"
server_key_iv: "869326a4a6a9de1de9bd7d353d7f1685"
server_key_version: "encryptionkey@0"
I20260621 02:44:43.556089 14856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:43.561512 14856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:43.562253 14856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:43.562520 14856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:43.563432 14856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:43.564780 14856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:43.564903 14856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:43.565029 14856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:43.565116 14856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:43.597532 14856 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:45679
I20260621 02:44:43.597579 14982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:45679 every 8 connection(s)
I20260621 02:44:43.598788 14856 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:44:43.600615 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 14856
I20260621 02:44:43.600775 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:44:43.601235 9880 external_mini_cluster.cc:1428] Setting key 9b3dfe9e0b5f35b143ffa3f6ee5ff8b7
I20260621 02:44:43.607120 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:43.601548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:47293 (local address 127.9.166.62:41399)
0621 02:44:43.601776 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:43.601782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:43.603319 (+ 1537us) server_negotiation.cc:407] Connection header received
0621 02:44:43.605064 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:43.605069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:43.605134 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:43.605244 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:43.606400 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:43.606405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:43.606410 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:44:43.606430 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:43.606467 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:44:43.606487 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:43.606610 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:43.606797 (+ 187us) server_negotiation.cc:299] Negotiation successful
0621 02:44:43.606868 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 02:44:43.608038 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:43.602140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:41399 (local address 127.9.166.3:47293)
0621 02:44:43.603182 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:43.603223 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:43.604825 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:43.605481 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:43.605489 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:43.605575 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:43.606135 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:44:43.606138 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:43.606172 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:44:43.606247 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:43.606577 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:43.606586 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:43.606596 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 02:44:43.606864 (+ 268us) client_negotiation.cc:240] Negotiation successful
0621 02:44:43.607087 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":344,"threads_started":1}
I20260621 02:44:43.609246 14983 heartbeater.cc:347] Connected to a master server at 127.9.166.62:41399
I20260621 02:44:43.609534 14983 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:43.610070 14983 heartbeater.cc:511] Master 127.9.166.62:41399 requested a full tablet report, sending...
I20260621 02:44:43.611327 14537 ts_manager.cc:194] Registered new tserver with Master: 19f720c0d7b143ce88e862d856b79559 (127.9.166.3:45679)
I20260621 02:44:43.616256 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:44:43.623003 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:43.620598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.9.166.62:41399)
0621 02:44:43.620779 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:43.620788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:43.621284 (+ 496us) server_negotiation.cc:407] Connection header received
0621 02:44:43.621385 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:43.621389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:43.621433 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:43.621511 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:43.622453 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:43.622457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:43.622460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:44:43.622473 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:43.622501 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:44:43.622512 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:43.622604 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:43.622672 (+ 68us) server_negotiation.cc:299] Negotiation successful
0621 02:44:43.622711 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:44:43.626725 14537 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:44:43.633473 14537 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:44:43.652473 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:43.645947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35910 (local address 127.9.166.3:45679)
0621 02:44:43.647937 (+ 1990us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:43.647950 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:43.648857 (+ 907us) server_negotiation.cc:407] Connection header received
0621 02:44:43.648950 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:43.648963 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:43.649233 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:43.649419 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:43.651555 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:43.651564 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:43.651578 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:44:43.651601 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:43.651651 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:44:43.651666 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:43.651840 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:43.651952 (+ 112us) server_negotiation.cc:299] Negotiation successful
0621 02:44:43.652050 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1776,"thread_start_us":1585,"threads_started":1}
I20260621 02:44:43.652527 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:43.645486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:45679 (local address 127.0.0.1:35910)
0621 02:44:43.648533 (+ 3047us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:43.648562 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:43.648701 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:43.650577 (+ 1876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:43.650583 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:43.650608 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:43.651291 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:44:43.651296 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:43.651315 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:44:43.651363 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:43.651762 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:43.651772 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:43.651780 (+ 8us) client_negotiation.cc:769] Sending connection context
0621 02:44:43.651946 (+ 166us) client_negotiation.cc:240] Negotiation successful
0621 02:44:43.652038 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2796,"thread_start_us":2619,"threads_started":1}
I20260621 02:44:43.652631 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:43.647963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48586 (local address 127.9.166.1:40939)
0621 02:44:43.648610 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:43.648618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:43.649158 (+ 540us) server_negotiation.cc:407] Connection header received
0621 02:44:43.649407 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:43.649416 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:43.649628 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:43.649748 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:43.651717 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:43.651725 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:43.651740 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:44:43.651764 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:43.651833 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:44:43.651855 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:43.651959 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:43.652206 (+ 247us) server_negotiation.cc:299] Negotiation successful
0621 02:44:43.652300 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":495,"threads_started":1}
I20260621 02:44:43.652726 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:43.645261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:42121 (local address 127.0.0.1:53190)
0621 02:44:43.646215 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:43.646271 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:43.646467 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:43.649836 (+ 3369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:43.649845 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:43.649947 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:43.650782 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:44:43.650787 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:43.650822 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:44:43.650932 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:43.652042 (+ 1110us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:43.652047 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:43.652052 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 02:44:43.652184 (+ 132us) client_negotiation.cc:240] Negotiation successful
0621 02:44:43.652237 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":409,"threads_started":1}
I20260621 02:44:43.653473 15000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:43.648275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:40939 (local address 127.0.0.1:48586)
0621 02:44:43.649031 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:43.649057 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:43.649244 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:43.650714 (+ 1470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:43.650724 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:43.650749 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:43.651490 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:44:43.651493 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:43.651509 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:44:43.651547 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:43.652042 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:43.652047 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:43.652052 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 02:44:43.652994 (+ 942us) client_negotiation.cc:240] Negotiation successful
0621 02:44:43.653040 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":76,"thread_start_us":398,"threads_started":1}
I20260621 02:44:43.653653 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:43.645514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53190 (local address 127.9.166.2:42121)
0621 02:44:43.649376 (+ 3862us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:43.649383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:43.649407 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:44:43.649464 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:43.649470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:43.649627 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:43.650408 (+ 781us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:43.651820 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:43.651828 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:43.651838 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:44:43.651860 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:43.651907 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:44:43.651925 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:43.653158 (+ 1233us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:43.653233 (+ 75us) server_negotiation.cc:299] Negotiation successful
0621 02:44:43.653337 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3646,"thread_start_us":3514,"threads_started":1}
I20260621 02:44:43.656843 14788 tablet_service.cc:1467] Processing CreateTablet for tablet c29a307c269f4d92880d624175019bcc (DEFAULT_TABLE table=test-table [id=fe3e724fa3b94ac38e5e0c89cbfe0c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:43.656960 14918 tablet_service.cc:1467] Processing CreateTablet for tablet c29a307c269f4d92880d624175019bcc (DEFAULT_TABLE table=test-table [id=fe3e724fa3b94ac38e5e0c89cbfe0c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:43.656960 14658 tablet_service.cc:1467] Processing CreateTablet for tablet c29a307c269f4d92880d624175019bcc (DEFAULT_TABLE table=test-table [id=fe3e724fa3b94ac38e5e0c89cbfe0c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:43.658281 14918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c29a307c269f4d92880d624175019bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:43.658283 14658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c29a307c269f4d92880d624175019bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:43.658294 14788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c29a307c269f4d92880d624175019bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:43.669493 15001 tablet_bootstrap.cc:492] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559: Bootstrap starting.
I20260621 02:44:43.673184 15002 tablet_bootstrap.cc:492] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0: Bootstrap starting.
I20260621 02:44:43.673668 15001 tablet_bootstrap.cc:654] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:43.674693 15003 tablet_bootstrap.cc:492] T c29a307c269f4d92880d624175019bcc P 18af4b546b3b417fa86bb470c50222c5: Bootstrap starting.
I20260621 02:44:43.675431 15001 log.cc:824] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:43.676637 15002 tablet_bootstrap.cc:654] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:43.677819 15002 log.cc:824] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:43.677995 15003 tablet_bootstrap.cc:654] T c29a307c269f4d92880d624175019bcc P 18af4b546b3b417fa86bb470c50222c5: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:43.678920 15001 tablet_bootstrap.cc:492] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559: No bootstrap required, opened a new log
I20260621 02:44:43.679188 15003 log.cc:824] T c29a307c269f4d92880d624175019bcc P 18af4b546b3b417fa86bb470c50222c5: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:43.679209 15001 ts_tablet_manager.cc:1397] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260621 02:44:43.680689 15002 tablet_bootstrap.cc:492] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0: No bootstrap required, opened a new log
I20260621 02:44:43.680927 15002 ts_tablet_manager.cc:1397] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 02:44:43.681527 15003 tablet_bootstrap.cc:492] T c29a307c269f4d92880d624175019bcc P 18af4b546b3b417fa86bb470c50222c5: No bootstrap required, opened a new log
I20260621 02:44:43.681758 15003 ts_tablet_manager.cc:1397] T c29a307c269f4d92880d624175019bcc P 18af4b546b3b417fa86bb470c50222c5: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260621 02:44:43.689060 15003 raft_consensus.cc:348] T c29a307c269f4d92880d624175019bcc P 18af4b546b3b417fa86bb470c50222c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd1db50076741e5b3f0e1e84f3e33f0" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42121 } } peers { permanent_uuid: "19f720c0d7b143ce88e862d856b79559" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 45679 } } peers { permanent_uuid: "18af4b546b3b417fa86bb470c50222c5" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 40939 } }
I20260621 02:44:43.689427 15003 raft_consensus.cc:374] T c29a307c269f4d92880d624175019bcc P 18af4b546b3b417fa86bb470c50222c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:43.689534 15003 raft_consensus.cc:729] T c29a307c269f4d92880d624175019bcc P 18af4b546b3b417fa86bb470c50222c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18af4b546b3b417fa86bb470c50222c5, State: Initialized, Role: FOLLOWER
I20260621 02:44:43.689764 15001 raft_consensus.cc:348] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd1db50076741e5b3f0e1e84f3e33f0" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42121 } } peers { permanent_uuid: "19f720c0d7b143ce88e862d856b79559" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 45679 } } peers { permanent_uuid: "18af4b546b3b417fa86bb470c50222c5" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 40939 } }
I20260621 02:44:43.690212 15001 raft_consensus.cc:374] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:43.690312 15001 raft_consensus.cc:729] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f720c0d7b143ce88e862d856b79559, State: Initialized, Role: FOLLOWER
I20260621 02:44:43.690182 15003 consensus_queue.cc:260] T c29a307c269f4d92880d624175019bcc P 18af4b546b3b417fa86bb470c50222c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd1db50076741e5b3f0e1e84f3e33f0" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42121 } } peers { permanent_uuid: "19f720c0d7b143ce88e862d856b79559" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 45679 } } peers { permanent_uuid: "18af4b546b3b417fa86bb470c50222c5" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 40939 } }
I20260621 02:44:43.690979 15001 consensus_queue.cc:260] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd1db50076741e5b3f0e1e84f3e33f0" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42121 } } peers { permanent_uuid: "19f720c0d7b143ce88e862d856b79559" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 45679 } } peers { permanent_uuid: "18af4b546b3b417fa86bb470c50222c5" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 40939 } }
I20260621 02:44:43.691058 15002 raft_consensus.cc:348] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd1db50076741e5b3f0e1e84f3e33f0" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42121 } } peers { permanent_uuid: "19f720c0d7b143ce88e862d856b79559" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 45679 } } peers { permanent_uuid: "18af4b546b3b417fa86bb470c50222c5" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 40939 } }
I20260621 02:44:43.691541 15002 raft_consensus.cc:374] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:43.691669 15002 raft_consensus.cc:729] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd1db50076741e5b3f0e1e84f3e33f0, State: Initialized, Role: FOLLOWER
I20260621 02:44:43.692364 15002 consensus_queue.cc:260] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd1db50076741e5b3f0e1e84f3e33f0" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42121 } } peers { permanent_uuid: "19f720c0d7b143ce88e862d856b79559" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 45679 } } peers { permanent_uuid: "18af4b546b3b417fa86bb470c50222c5" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 40939 } }
I20260621 02:44:43.693833 14853 heartbeater.cc:503] Master 127.9.166.62:41399 was elected leader, sending a full tablet report...
I20260621 02:44:43.696353 15002 ts_tablet_manager.cc:1428] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0: Time spent starting tablet: real 0.015s user 0.006s sys 0.008s
I20260621 02:44:43.697500 14723 heartbeater.cc:503] Master 127.9.166.62:41399 was elected leader, sending a full tablet report...
I20260621 02:44:43.698017 15003 ts_tablet_manager.cc:1428] T c29a307c269f4d92880d624175019bcc P 18af4b546b3b417fa86bb470c50222c5: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
I20260621 02:44:43.701149 14983 heartbeater.cc:503] Master 127.9.166.62:41399 was elected leader, sending a full tablet report...
I20260621 02:44:43.701783 15001 ts_tablet_manager.cc:1428] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559: Time spent starting tablet: real 0.022s user 0.015s sys 0.007s
W20260621 02:44:43.753512 14724 tablet.cc:2369] T c29a307c269f4d92880d624175019bcc P 18af4b546b3b417fa86bb470c50222c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:44:43.850502 14984 tablet.cc:2369] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:44:43.920012 14854 tablet.cc:2369] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:44:44.126394 15009 raft_consensus.cc:482] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:44:44.126700 15009 raft_consensus.cc:504] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd1db50076741e5b3f0e1e84f3e33f0" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42121 } } peers { permanent_uuid: "19f720c0d7b143ce88e862d856b79559" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 45679 } } peers { permanent_uuid: "18af4b546b3b417fa86bb470c50222c5" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 40939 } }
I20260621 02:44:44.128486 15009 leader_election.cc:288] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19f720c0d7b143ce88e862d856b79559 (127.9.166.3:45679), 18af4b546b3b417fa86bb470c50222c5 (127.9.166.1:40939)
I20260621 02:44:44.132074 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:44.128864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:52795 (local address 127.9.166.1:40939)
0621 02:44:44.129104 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:44.129110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:44.129876 (+ 766us) server_negotiation.cc:407] Connection header received
0621 02:44:44.130078 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:44.130085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:44.130154 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:44.130259 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:44.131278 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:44.131284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:44.131289 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:44:44.131307 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:44.131357 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:44:44.131369 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:44.131518 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:44.131707 (+ 189us) server_negotiation.cc:299] Negotiation successful
0621 02:44:44.131773 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 02:44:44.132148 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:44.128879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:46723 (local address 127.9.166.3:45679)
0621 02:44:44.129068 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:44.129082 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:44.130068 (+ 986us) server_negotiation.cc:407] Connection header received
0621 02:44:44.130279 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:44.130282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:44.130328 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:44.130436 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:44.131467 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:44.131473 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:44.131477 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:44:44.131495 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:44.131534 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:44:44.131548 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:44.131658 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:44.131848 (+ 190us) server_negotiation.cc:299] Negotiation successful
0621 02:44:44.131912 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 02:44:44.132205 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:44.128869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:40939 (local address 127.9.166.2:52795)
0621 02:44:44.129767 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:44.129784 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:44.129937 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:44.130282 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:44.130286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:44.130308 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:44.131051 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:44:44.131054 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:44.131067 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:44:44.131106 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:44.131569 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:44.131574 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:44.131579 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 02:44:44.131695 (+ 116us) client_negotiation.cc:240] Negotiation successful
0621 02:44:44.131745 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":795,"threads_started":1}
I20260621 02:44:44.132442 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:44.129137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:45679 (local address 127.9.166.2:46723)
0621 02:44:44.129969 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:44.129988 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:44.130143 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:44.130458 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:44.130463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:44.130485 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:44.131257 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:44:44.131260 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:44.131272 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:44:44.131305 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:44.131708 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:44.131713 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:44.131719 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 02:44:44.131857 (+ 138us) client_negotiation.cc:240] Negotiation successful
0621 02:44:44.131908 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":349,"threads_started":1}
I20260621 02:44:44.133443 14938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c29a307c269f4d92880d624175019bcc" candidate_uuid: "ffd1db50076741e5b3f0e1e84f3e33f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f720c0d7b143ce88e862d856b79559" is_pre_election: true
I20260621 02:44:44.133443 14678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c29a307c269f4d92880d624175019bcc" candidate_uuid: "ffd1db50076741e5b3f0e1e84f3e33f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18af4b546b3b417fa86bb470c50222c5" is_pre_election: true
I20260621 02:44:44.133957 14938 raft_consensus.cc:2454] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffd1db50076741e5b3f0e1e84f3e33f0 in term 0.
I20260621 02:44:44.133957 14678 raft_consensus.cc:2454] T c29a307c269f4d92880d624175019bcc P 18af4b546b3b417fa86bb470c50222c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffd1db50076741e5b3f0e1e84f3e33f0 in term 0.
I20260621 02:44:44.134620 14743 leader_election.cc:302] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18af4b546b3b417fa86bb470c50222c5, ffd1db50076741e5b3f0e1e84f3e33f0; no voters:
I20260621 02:44:44.135010 15009 raft_consensus.cc:2781] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:44:44.135200 15009 raft_consensus.cc:482] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:44:44.135305 15009 raft_consensus.cc:3037] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:44.137521 15009 raft_consensus.cc:504] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd1db50076741e5b3f0e1e84f3e33f0" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42121 } } peers { permanent_uuid: "19f720c0d7b143ce88e862d856b79559" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 45679 } } peers { permanent_uuid: "18af4b546b3b417fa86bb470c50222c5" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 40939 } }
I20260621 02:44:44.138098 15009 leader_election.cc:288] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [CANDIDATE]: Term 1 election: Requested vote from peers 19f720c0d7b143ce88e862d856b79559 (127.9.166.3:45679), 18af4b546b3b417fa86bb470c50222c5 (127.9.166.1:40939)
I20260621 02:44:44.138518 14678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c29a307c269f4d92880d624175019bcc" candidate_uuid: "ffd1db50076741e5b3f0e1e84f3e33f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18af4b546b3b417fa86bb470c50222c5"
I20260621 02:44:44.138530 14938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c29a307c269f4d92880d624175019bcc" candidate_uuid: "ffd1db50076741e5b3f0e1e84f3e33f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f720c0d7b143ce88e862d856b79559"
I20260621 02:44:44.138711 14678 raft_consensus.cc:3037] T c29a307c269f4d92880d624175019bcc P 18af4b546b3b417fa86bb470c50222c5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:44.138721 14938 raft_consensus.cc:3037] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:44.140841 14938 raft_consensus.cc:2454] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffd1db50076741e5b3f0e1e84f3e33f0 in term 1.
I20260621 02:44:44.140944 14678 raft_consensus.cc:2454] T c29a307c269f4d92880d624175019bcc P 18af4b546b3b417fa86bb470c50222c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffd1db50076741e5b3f0e1e84f3e33f0 in term 1.
I20260621 02:44:44.141275 14744 leader_election.cc:302] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f720c0d7b143ce88e862d856b79559, ffd1db50076741e5b3f0e1e84f3e33f0; no voters:
I20260621 02:44:44.141566 15009 raft_consensus.cc:2781] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:44.142019 15009 raft_consensus.cc:686] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [term 1 LEADER]: Becoming Leader. State: Replica: ffd1db50076741e5b3f0e1e84f3e33f0, State: Running, Role: LEADER
I20260621 02:44:44.142501 15009 consensus_queue.cc:237] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd1db50076741e5b3f0e1e84f3e33f0" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42121 } } peers { permanent_uuid: "19f720c0d7b143ce88e862d856b79559" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 45679 } } peers { permanent_uuid: "18af4b546b3b417fa86bb470c50222c5" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 40939 } }
I20260621 02:44:44.146664 14537 catalog_manager.cc:5495] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffd1db50076741e5b3f0e1e84f3e33f0 (127.9.166.2). New cstate: current_term: 1 leader_uuid: "ffd1db50076741e5b3f0e1e84f3e33f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffd1db50076741e5b3f0e1e84f3e33f0" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 42121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19f720c0d7b143ce88e862d856b79559" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 45679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18af4b546b3b417fa86bb470c50222c5" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 40939 } health_report { overall_health: UNKNOWN } } }
I20260621 02:44:44.220538 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:44.217527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.9.166.2:42121)
0621 02:44:44.217987 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:44.217992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:44.218400 (+ 408us) server_negotiation.cc:407] Connection header received
0621 02:44:44.218618 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:44.218623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:44.218711 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:44.218828 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:44.219837 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:44.219842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:44.219853 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:44:44.219869 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:44.219907 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:44:44.219918 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:44.220040 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:44.220149 (+ 109us) server_negotiation.cc:299] Negotiation successful
0621 02:44:44.220207 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":228,"threads_started":1}
I20260621 02:44:44.228392 14678 raft_consensus.cc:1261] T c29a307c269f4d92880d624175019bcc P 18af4b546b3b417fa86bb470c50222c5 [term 1 FOLLOWER]: Refusing update from remote peer ffd1db50076741e5b3f0e1e84f3e33f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:44:44.228444 14938 raft_consensus.cc:1261] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559 [term 1 FOLLOWER]: Refusing update from remote peer ffd1db50076741e5b3f0e1e84f3e33f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:44:44.229319 15012 consensus_queue.cc:1035] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f720c0d7b143ce88e862d856b79559" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 45679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:44.229645 15009 consensus_queue.cc:1035] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18af4b546b3b417fa86bb470c50222c5" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 40939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:44.252664 14536 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:44:44.252924 14536 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:44:44.256415 14536 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d22cf058abf9409b94c06ad1a38dd85f: Sending DeleteTablet for 3 replicas of tablet c29a307c269f4d92880d624175019bcc
I20260621 02:44:44.257495 14788 tablet_service.cc:1514] Processing DeleteTablet for tablet c29a307c269f4d92880d624175019bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:44 UTC) from {username='slave'} at 127.0.0.1:53190
I20260621 02:44:44.257494 14918 tablet_service.cc:1514] Processing DeleteTablet for tablet c29a307c269f4d92880d624175019bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:44 UTC) from {username='slave'} at 127.0.0.1:35910
I20260621 02:44:44.261075 14658 tablet_service.cc:1514] Processing DeleteTablet for tablet c29a307c269f4d92880d624175019bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:44:44 UTC) from {username='slave'} at 127.0.0.1:48586
I20260621 02:44:44.263044 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 14596
I20260621 02:44:44.274322 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:44.264788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48588 (local address 127.9.166.1:40939)
0621 02:44:44.264996 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:44.265004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:44.271884 (+ 6880us) server_negotiation.cc:407] Connection header received
0621 02:44:44.272154 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:44.272160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:44.272269 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:44.272346 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:44.273572 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:44.273578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:44.273583 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:44:44.273601 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:44.273639 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:44:44.273654 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:44.273745 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:44.273975 (+ 230us) server_negotiation.cc:299] Negotiation successful
0621 02:44:44.274035 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:44:44.275333 14718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:44.283124 15031 tablet_replica.cc:330] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559: stopping tablet replica
I20260621 02:44:44.283412 15032 tablet_replica.cc:330] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0: stopping tablet replica
I20260621 02:44:44.284101 15031 raft_consensus.cc:2229] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:44:44.284615 15031 raft_consensus.cc:2258] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:44.285073 15032 raft_consensus.cc:2229] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:44:44.285772 15032 raft_consensus.cc:2258] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:44:44.288129 15031 ts_tablet_manager.cc:1905] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:44.293514 15032 ts_tablet_manager.cc:1905] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:44.296224 15031 ts_tablet_manager.cc:1918] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:44:44.296437 15031 log.cc:1196] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/c29a307c269f4d92880d624175019bcc
I20260621 02:44:44.296990 15031 ts_tablet_manager.cc:1939] T c29a307c269f4d92880d624175019bcc P 19f720c0d7b143ce88e862d856b79559: Deleting consensus metadata
I20260621 02:44:44.299448 14524 catalog_manager.cc:4841] TS 19f720c0d7b143ce88e862d856b79559 (127.9.166.3:45679): tablet c29a307c269f4d92880d624175019bcc (table test-table [id=fe3e724fa3b94ac38e5e0c89cbfe0c2c]) successfully deleted
I20260621 02:44:44.300854 15032 ts_tablet_manager.cc:1918] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:44:44.301083 15032 log.cc:1196] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/c29a307c269f4d92880d624175019bcc
I20260621 02:44:44.301652 15032 ts_tablet_manager.cc:1939] T c29a307c269f4d92880d624175019bcc P ffd1db50076741e5b3f0e1e84f3e33f0: Deleting consensus metadata
I20260621 02:44:44.303231 14523 catalog_manager.cc:4841] TS ffd1db50076741e5b3f0e1e84f3e33f0 (127.9.166.2:42121): tablet c29a307c269f4d92880d624175019bcc (table test-table [id=fe3e724fa3b94ac38e5e0c89cbfe0c2c]) successfully deleted
I20260621 02:44:44.373270 15033 tablet_replica.cc:330] T c29a307c269f4d92880d624175019bcc P 18af4b546b3b417fa86bb470c50222c5: stopping tablet replica
I20260621 02:44:44.373528 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 14596
W20260621 02:44:44.392822 14523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.166.1:40939 (error 108)
I20260621 02:44:44.393085 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 14726
I20260621 02:44:44.395435 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:44.394429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:40939 (local address 127.0.0.1:48594)
0621 02:44:44.395021 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:44.395187 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":340,"threads_started":1}
W20260621 02:44:44.395728 14523 catalog_manager.cc:4584] TS 18af4b546b3b417fa86bb470c50222c5 (127.9.166.1:40939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c29a307c269f4d92880d624175019bcc: Network error: Client connection negotiation failed: client connection to 127.9.166.1:40939: connect: Connection refused (error 111)
I20260621 02:44:44.396481 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:44.394013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53208 (local address 127.9.166.2:42121)
0621 02:44:44.394186 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:44.394193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:44.394213 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:44:44.394366 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:44.394372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:44.394435 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:44.394547 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:44.395573 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:44.395580 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:44.395589 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:44:44.395611 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:44.395649 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:44:44.395666 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:44.395792 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:44.396213 (+ 421us) server_negotiation.cc:299] Negotiation successful
0621 02:44:44.396278 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:44:44.397260 14848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:44.449429 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:44.448989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:40939 (local address 127.0.0.1:48600)
0621 02:44:44.449207 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:44.449289 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260621 02:44:44.450529 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:44.450185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:40939 (local address 127.0.0.1:48616)
0621 02:44:44.450343 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:44.450406 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 02:44:44.503450 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 14726
I20260621 02:44:44.510936 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:44.510508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:40939 (local address 127.0.0.1:48630)
0621 02:44:44.510709 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:44.510797 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260621 02:44:44.511986 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:44.511609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:40939 (local address 127.0.0.1:48636)
0621 02:44:44.511755 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:44.511846 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260621 02:44:44.521816 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 14856
I20260621 02:44:44.524652 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:44.522529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35926 (local address 127.9.166.3:45679)
0621 02:44:44.522716 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:44.522722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:44.522742 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:44:44.522926 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:44.522931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:44.523044 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:44.523152 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:44.524041 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:44.524047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:44.524051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:44:44.524071 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:44.524106 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:44:44.524118 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:44.524243 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:44.524324 (+ 81us) server_negotiation.cc:299] Negotiation successful
0621 02:44:44.524380 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:44:44.525300 14978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:44.592396 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:44.591783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:40939 (local address 127.0.0.1:48638)
0621 02:44:44.592079 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:44.592191 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260621 02:44:44.593573 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:44.593113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:40939 (local address 127.0.0.1:48650)
0621 02:44:44.593315 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:44.593415 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260621 02:44:44.611742 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 14856
I20260621 02:44:44.628005 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 14506
I20260621 02:44:44.629117 14567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:44.735518 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 14506
2026-06-21T02:44:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2946 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260621 02:44:44.775993 9880 mini_oidc.cc:152] Starting JWKS server
I20260621 02:44:44.778447 9880 webserver.cc:460] Webserver started at https://127.0.0.1:33501/ using document root <none> and password file <none>
I20260621 02:44:44.778756 9880 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 02:44:44.779374 9880 webserver.cc:460] Webserver started at http://127.0.0.1:42271/ using document root <none> and password file <none>
2026-06-21T02:44:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:44:44Z Disabled control of system clock
I20260621 02:44:44.805086 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:46047
--webserver_interface=127.9.166.62
--webserver_port=0
--builtin_ntp_servers=127.9.166.20:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:46047
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33501/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009844354615-9880-0/testchainca.cert with env {}
W20260621 02:44:45.120574 15046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:45.120848 15046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:45.120939 15046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:45.129108 15046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:44:45.129230 15046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:45.129287 15046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:45.129348 15046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:44:45.129390 15046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 02:44:45.129477 15046 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 02:44:45.129522 15046 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33501/jwks.json
I20260621 02:44:45.140758 15046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:46047
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:46047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33501/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009844354615-9880-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:45.142056 15046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:45.143378 15046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:45.155820 15052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:45.155920 15051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:45.155792 15054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:45.156626 15046 server_base.cc:1034] running on GCE node
I20260621 02:44:45.157382 15046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:45.158715 15046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:45.159937 15046 hybrid_clock.cc:648] HybridClock initialized: now 1782009885159896 us; error 55 us; skew 500 ppm
I20260621 02:44:45.160271 15046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:44:45.162426 15046 webserver.cc:460] Webserver started at http://127.9.166.62:35935/ using document root <none> and password file <none>
I20260621 02:44:45.163090 15046 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:45.163220 15046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:45.163594 15046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:45.166370 15046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "fc69bbe57c9a45f1949853215d1834ad"
format_stamp: "Formatted at 2026-06-21 02:44:45 on dist-test-slave-hvgn"
server_key: "c90ee9e2bde2eb95094ee28e9dab25a5"
server_key_iv: "8969b38d26aa80aeadfb9b48c95e2053"
server_key_version: "encryptionkey@0"
I20260621 02:44:45.167135 15046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "fc69bbe57c9a45f1949853215d1834ad"
format_stamp: "Formatted at 2026-06-21 02:44:45 on dist-test-slave-hvgn"
server_key: "c90ee9e2bde2eb95094ee28e9dab25a5"
server_key_iv: "8969b38d26aa80aeadfb9b48c95e2053"
server_key_version: "encryptionkey@0"
I20260621 02:44:45.171981 15046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 02:44:45.175199 15060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:45.176484 15046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:44:45.176671 15046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "fc69bbe57c9a45f1949853215d1834ad"
format_stamp: "Formatted at 2026-06-21 02:44:45 on dist-test-slave-hvgn"
server_key: "c90ee9e2bde2eb95094ee28e9dab25a5"
server_key_iv: "8969b38d26aa80aeadfb9b48c95e2053"
server_key_version: "encryptionkey@0"
I20260621 02:44:45.176872 15046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:45.205570 15046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:45.206348 15046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:45.206655 15046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:45.224536 15046 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:46047
I20260621 02:44:45.224538 15111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:46047 every 8 connection(s)
I20260621 02:44:45.225911 15046 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:44:45.229437 15112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:45.231786 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 15046
I20260621 02:44:45.232000 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:44:45.232632 9880 external_mini_cluster.cc:1428] Setting key e324c3c897c8c1bf2364c8a4b7810f8f
I20260621 02:44:45.242139 15112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad: Bootstrap starting.
I20260621 02:44:45.245775 15112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:45.247036 15112 log.cc:824] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:45.247862 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:45.234537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.9.166.62:46047)
0621 02:44:45.235539 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:45.235553 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:45.235602 (+ 49us) server_negotiation.cc:407] Connection header received
0621 02:44:45.237857 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:45.237888 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:45.238488 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:45.238903 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:45.239458 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:45.241406 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:45.242749 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:45.243826 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:45.244004 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:45.245134 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:45.245144 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:45.245160 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:44:45.245836 (+ 676us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:45.245946 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:44:45.245973 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:45.246102 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:45.246305 (+ 203us) server_negotiation.cc:299] Negotiation successful
0621 02:44:45.246587 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":281,"threads_started":1}
I20260621 02:44:45.249775 15112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad: No bootstrap required, opened a new log
I20260621 02:44:45.251457 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:44:45.257414 15112 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc69bbe57c9a45f1949853215d1834ad" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46047 } }
I20260621 02:44:45.257856 15112 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:45.257962 15112 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc69bbe57c9a45f1949853215d1834ad, State: Initialized, Role: FOLLOWER
I20260621 02:44:45.258494 15112 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc69bbe57c9a45f1949853215d1834ad" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46047 } }
I20260621 02:44:45.258740 15112 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:44:45.258875 15112 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:44:45.259050 15112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:45.261152 15112 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc69bbe57c9a45f1949853215d1834ad" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46047 } }
I20260621 02:44:45.261677 15112 leader_election.cc:302] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc69bbe57c9a45f1949853215d1834ad; no voters:
I20260621 02:44:45.262219 15112 leader_election.cc:288] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:44:45.262360 15117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:45.263024 15117 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad [term 1 LEADER]: Becoming Leader. State: Replica: fc69bbe57c9a45f1949853215d1834ad, State: Running, Role: LEADER
I20260621 02:44:45.263723 15117 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc69bbe57c9a45f1949853215d1834ad" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46047 } }
I20260621 02:44:45.264307 15112 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:44:45.266814 15119 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc69bbe57c9a45f1949853215d1834ad. Latest consensus state: current_term: 1 leader_uuid: "fc69bbe57c9a45f1949853215d1834ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc69bbe57c9a45f1949853215d1834ad" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46047 } } }
I20260621 02:44:45.267786 15119 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:45.266739 15118 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc69bbe57c9a45f1949853215d1834ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc69bbe57c9a45f1949853215d1834ad" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46047 } } }
I20260621 02:44:45.274065 15118 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:45.276494 15123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:44:45.281905 15123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:44:45.288461 15134 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 02:44:45.292201 15123 catalog_manager.cc:1347] Generated new cluster ID: b967593c25d14dca96d5cb1e0db2e302
I20260621 02:44:45.292296 15123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:44:45.316501 15123 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:44:45.318280 15123 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:44:45.330754 15123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc69bbe57c9a45f1949853215d1834ad: Generated new TSK 0
I20260621 02:44:45.331636 15123 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260621 02:44:45.371508 15142 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260621 02:44:45.371692 15142 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 02:44:45.372839 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:45.351318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.9.166.62:46047)
0621 02:44:45.351513 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:45.351520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:45.351979 (+ 459us) server_negotiation.cc:407] Connection header received
0621 02:44:45.352259 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:45.352265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:45.352357 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:45.352509 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0621 02:44:45.352970 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:45.354123 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:45.355538 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:45.356246 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:45.356368 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:45.356846 (+ 478us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 02:44:45.372166 (+ 15320us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0621 02:44:45.372532 (+ 366us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53374: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260621 02:44:45.372956 15115 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53374: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260621 02:44:45.374109 15141 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:44:45.351249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:46047 (local address 127.0.0.1:53374)
0621 02:44:45.351883 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:45.351903 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:45.352040 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:45.352530 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:45.352533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:45.352542 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0621 02:44:45.352839 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:45.352847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:45.354244 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:45.354247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:45.355359 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:45.355369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:45.356416 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:45.356421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:45.356539 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:45.356592 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 02:44:45.372435 (+ 15843us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0621 02:44:45.372697 (+ 262us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.166.62:46047: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":358,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.9.166.62:46047: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33501/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260621 02:44:45.376296 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 15046
I20260621 02:44:45.377442 15107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:45.470098 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 15046
2026-06-21T02:44:45Z chronyd exiting
I20260621 02:44:45.519593 9880 test_util.cc:175] -----------------------------------------------
I20260621 02:44:45.519749 9880 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1782009844354615-9880-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (747 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260621 02:44:45.523330 9880 mini_oidc.cc:152] Starting JWKS server
I20260621 02:44:45.525101 9880 webserver.cc:460] Webserver started at https://127.0.0.1:41887/ using document root <none> and password file <none>
I20260621 02:44:45.525333 9880 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 02:44:45.526070 9880 webserver.cc:460] Webserver started at http://127.0.0.1:36779/ using document root <none> and password file <none>
2026-06-21T02:44:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:44:45Z Disabled control of system clock
I20260621 02:44:45.551640 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:46725
--webserver_interface=127.9.166.62
--webserver_port=0
--builtin_ntp_servers=127.9.166.20:37471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:46725
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41887/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009844354615-9880-0/testchainca.cert with env {}
W20260621 02:44:45.863736 15149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:45.864014 15149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:45.864099 15149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:45.871116 15149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:44:45.871233 15149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:45.871270 15149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:45.871328 15149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:44:45.871366 15149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 02:44:45.871485 15149 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 02:44:45.871533 15149 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41887/jwks.json
I20260621 02:44:45.881196 15149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:37471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:46725
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:46725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41887/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009844354615-9880-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:45.882369 15149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:45.883610 15149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:45.895191 15154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:45.895367 15155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:45.895376 15157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:45.896301 15149 server_base.cc:1034] running on GCE node
I20260621 02:44:45.897095 15149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:45.898360 15149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:45.899549 15149 hybrid_clock.cc:648] HybridClock initialized: now 1782009885899487 us; error 58 us; skew 500 ppm
I20260621 02:44:45.899890 15149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:44:45.907008 15149 webserver.cc:460] Webserver started at http://127.9.166.62:36671/ using document root <none> and password file <none>
I20260621 02:44:45.907771 15149 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:45.907892 15149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:45.908223 15149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:45.910980 15149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "e9779b2b6ebe4c14a8833e5976cdfaf2"
format_stamp: "Formatted at 2026-06-21 02:44:45 on dist-test-slave-hvgn"
server_key: "7508c1c3b0c69aaa830dc5a2fa756d3a"
server_key_iv: "38b32630a5c9c9380732c75e88c3b406"
server_key_version: "encryptionkey@0"
I20260621 02:44:45.911793 15149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "e9779b2b6ebe4c14a8833e5976cdfaf2"
format_stamp: "Formatted at 2026-06-21 02:44:45 on dist-test-slave-hvgn"
server_key: "7508c1c3b0c69aaa830dc5a2fa756d3a"
server_key_iv: "38b32630a5c9c9380732c75e88c3b406"
server_key_version: "encryptionkey@0"
I20260621 02:44:45.916553 15149 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 02:44:45.919852 15163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:45.921128 15149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:44:45.921325 15149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "e9779b2b6ebe4c14a8833e5976cdfaf2"
format_stamp: "Formatted at 2026-06-21 02:44:45 on dist-test-slave-hvgn"
server_key: "7508c1c3b0c69aaa830dc5a2fa756d3a"
server_key_iv: "38b32630a5c9c9380732c75e88c3b406"
server_key_version: "encryptionkey@0"
I20260621 02:44:45.921525 15149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:45.948073 15149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:45.948872 15149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:45.949184 15149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:45.967240 15149 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:46725
I20260621 02:44:45.967259 15214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:46725 every 8 connection(s)
I20260621 02:44:45.968529 15149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:44:45.972100 15215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:45.978030 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 15149
I20260621 02:44:45.978238 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:44:45.979038 9880 external_mini_cluster.cc:1428] Setting key 5f22ebe99aecb080a927ef88d05f4710
I20260621 02:44:45.982762 15215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2: Bootstrap starting.
I20260621 02:44:45.985965 15215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:45.987149 15215 log.cc:824] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:45.990188 15215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2: No bootstrap required, opened a new log
I20260621 02:44:45.992475 15218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:45.980960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35880 (local address 127.9.166.62:46725)
0621 02:44:45.981893 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:45.981903 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:45.981943 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:44:45.983699 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:45.983720 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:45.984159 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:45.984459 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:45.984999 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:45.986351 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:45.988125 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:45.988867 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:45.988989 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:45.990193 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:45.990200 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:45.990213 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:44:45.990779 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:45.990853 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:44:45.990875 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:45.990956 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:45.991198 (+ 242us) server_negotiation.cc:299] Negotiation successful
0621 02:44:45.991525 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":354,"threads_started":1}
I20260621 02:44:45.996273 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:44:45.997985 15215 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9779b2b6ebe4c14a8833e5976cdfaf2" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46725 } }
I20260621 02:44:45.998328 15215 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:45.998477 15215 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9779b2b6ebe4c14a8833e5976cdfaf2, State: Initialized, Role: FOLLOWER
I20260621 02:44:45.999004 15215 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9779b2b6ebe4c14a8833e5976cdfaf2" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46725 } }
I20260621 02:44:45.999229 15215 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:44:45.999428 15215 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:44:45.999610 15215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:46.001988 15215 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9779b2b6ebe4c14a8833e5976cdfaf2" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46725 } }
I20260621 02:44:46.002410 15215 leader_election.cc:302] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9779b2b6ebe4c14a8833e5976cdfaf2; no voters:
I20260621 02:44:46.002889 15215 leader_election.cc:288] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:44:46.003067 15220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:46.003904 15220 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2 [term 1 LEADER]: Becoming Leader. State: Replica: e9779b2b6ebe4c14a8833e5976cdfaf2, State: Running, Role: LEADER
I20260621 02:44:46.004403 15220 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9779b2b6ebe4c14a8833e5976cdfaf2" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46725 } }
I20260621 02:44:46.004897 15215 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:44:46.007159 15222 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9779b2b6ebe4c14a8833e5976cdfaf2. Latest consensus state: current_term: 1 leader_uuid: "e9779b2b6ebe4c14a8833e5976cdfaf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9779b2b6ebe4c14a8833e5976cdfaf2" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46725 } } }
I20260621 02:44:46.007185 15221 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9779b2b6ebe4c14a8833e5976cdfaf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9779b2b6ebe4c14a8833e5976cdfaf2" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46725 } } }
I20260621 02:44:46.007679 15221 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:46.007663 15222 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:46.009424 15225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:44:46.013703 15225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:44:46.022261 15225 catalog_manager.cc:1347] Generated new cluster ID: df9579a2039749538a6829b85cfcb8e4
I20260621 02:44:46.022362 15225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:44:46.035928 15225 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:44:46.037002 15225 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:44:46.047509 15225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9779b2b6ebe4c14a8833e5976cdfaf2: Generated new TSK 0
I20260621 02:44:46.048307 15225 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260621 02:44:46.089949 15245 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260621 02:44:46.090065 15245 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 02:44:46.091212 15218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:46.072007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35886 (local address 127.9.166.62:46725)
0621 02:44:46.072210 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:46.072216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:46.072597 (+ 381us) server_negotiation.cc:407] Connection header received
0621 02:44:46.072792 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:46.072798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:46.072905 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:46.073071 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0621 02:44:46.073482 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:46.074615 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:46.075958 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:46.076639 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:46.076763 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:46.077183 (+ 420us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 02:44:46.090529 (+ 13346us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41887/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0621 02:44:46.090927 (+ 398us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35886: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41887/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":65}
W20260621 02:44:46.091336 15218 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35886: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41887/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260621 02:44:46.091456 15244 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:44:46.071875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:46725 (local address 127.0.0.1:35886)
0621 02:44:46.072503 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:46.072518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:46.072630 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:46.073075 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:46.073078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:46.073087 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0621 02:44:46.073325 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:46.073332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:46.074739 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:46.074742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:46.075824 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:46.075835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:46.076792 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:46.076796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:46.076923 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:46.076952 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 02:44:46.090761 (+ 13809us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41887/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0621 02:44:46.091002 (+ 241us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.166.62:46725: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41887/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":362,"threads_started":1}
I20260621 02:44:46.093160 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 15149
I20260621 02:44:46.094178 15210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:46.185372 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 15149
2026-06-21T02:44:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (716 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260621 02:44:46.242192 9880 mini_oidc.cc:152] Starting JWKS server
I20260621 02:44:46.244910 9880 webserver.cc:460] Webserver started at https://127.0.0.1:39207/ using document root <none> and password file <none>
I20260621 02:44:46.245184 9880 mini_oidc.cc:189] Starting OIDC Discovery server
I20260621 02:44:46.245995 9880 webserver.cc:460] Webserver started at http://127.0.0.1:38039/ using document root <none> and password file <none>
2026-06-21T02:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:44:46Z Disabled control of system clock
I20260621 02:44:46.272291 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:37825
--webserver_interface=127.9.166.62
--webserver_port=0
--builtin_ntp_servers=127.9.166.20:42689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:37825
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39207/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260621 02:44:46.582288 15252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:46.582548 15252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:46.582639 15252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:46.589649 15252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:44:46.589740 15252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:46.589777 15252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:46.589829 15252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:44:46.589872 15252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 02:44:46.589923 15252 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 02:44:46.589962 15252 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39207/jwks.json
I20260621 02:44:46.598639 15252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:37825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:37825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39207/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:46.599817 15252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:46.601511 15252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:46.612653 15258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:46.612800 15257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:46.612823 15260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:46.613749 15252 server_base.cc:1034] running on GCE node
I20260621 02:44:46.614475 15252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:46.615943 15252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:46.617136 15252 hybrid_clock.cc:648] HybridClock initialized: now 1782009886617083 us; error 55 us; skew 500 ppm
I20260621 02:44:46.617463 15252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:44:46.619724 15252 webserver.cc:460] Webserver started at http://127.9.166.62:38251/ using document root <none> and password file <none>
I20260621 02:44:46.620371 15252 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:46.620481 15252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:46.620826 15252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:46.623593 15252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "fbc61ef52a5e4e9ca5740a9db793a40d"
format_stamp: "Formatted at 2026-06-21 02:44:46 on dist-test-slave-hvgn"
server_key: "f2a167b67b1974e38cbc2117e979338c"
server_key_iv: "85f38801d2ddf72148b072bba8ce3d6f"
server_key_version: "encryptionkey@0"
I20260621 02:44:46.624372 15252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "fbc61ef52a5e4e9ca5740a9db793a40d"
format_stamp: "Formatted at 2026-06-21 02:44:46 on dist-test-slave-hvgn"
server_key: "f2a167b67b1974e38cbc2117e979338c"
server_key_iv: "85f38801d2ddf72148b072bba8ce3d6f"
server_key_version: "encryptionkey@0"
I20260621 02:44:46.629251 15252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:44:46.632462 15266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:46.633661 15252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:44:46.633847 15252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "fbc61ef52a5e4e9ca5740a9db793a40d"
format_stamp: "Formatted at 2026-06-21 02:44:46 on dist-test-slave-hvgn"
server_key: "f2a167b67b1974e38cbc2117e979338c"
server_key_iv: "85f38801d2ddf72148b072bba8ce3d6f"
server_key_version: "encryptionkey@0"
I20260621 02:44:46.634045 15252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:46.654098 15252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:46.654886 15252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:46.655184 15252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:46.674237 15252 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:37825
I20260621 02:44:46.674259 15317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:37825 every 8 connection(s)
I20260621 02:44:46.675580 15252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:44:46.678207 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 15252
I20260621 02:44:46.678422 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:44:46.679045 9880 external_mini_cluster.cc:1428] Setting key d88b4d9c51335ec9a6960b3dc35319a6
I20260621 02:44:46.679340 15318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:46.692319 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:46.680927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41148 (local address 127.9.166.62:37825)
0621 02:44:46.681922 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:46.681932 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:46.681973 (+ 41us) server_negotiation.cc:407] Connection header received
0621 02:44:46.683782 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:46.683803 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:46.684231 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:46.684536 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:46.685105 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:46.686378 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:46.688192 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:46.688836 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:46.688971 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:46.690141 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:46.690148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:46.690159 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:44:46.690705 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:46.690784 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:44:46.690803 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:46.690892 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:46.691109 (+ 217us) server_negotiation.cc:299] Negotiation successful
0621 02:44:46.691456 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":570,"thread_start_us":358,"threads_started":1}
I20260621 02:44:46.692835 15318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d: Bootstrap starting.
I20260621 02:44:46.695991 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:44:46.696313 15318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:46.697243 15318 log.cc:824] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:46.700069 15318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d: No bootstrap required, opened a new log
I20260621 02:44:46.706913 15318 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc61ef52a5e4e9ca5740a9db793a40d" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37825 } }
I20260621 02:44:46.707234 15318 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:46.707345 15318 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc61ef52a5e4e9ca5740a9db793a40d, State: Initialized, Role: FOLLOWER
I20260621 02:44:46.707996 15318 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc61ef52a5e4e9ca5740a9db793a40d" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37825 } }
I20260621 02:44:46.708266 15318 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:44:46.708418 15318 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:44:46.708585 15318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:46.710566 15318 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc61ef52a5e4e9ca5740a9db793a40d" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37825 } }
I20260621 02:44:46.711055 15318 leader_election.cc:302] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbc61ef52a5e4e9ca5740a9db793a40d; no voters:
I20260621 02:44:46.711645 15318 leader_election.cc:288] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:44:46.711773 15323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:46.712973 15323 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d [term 1 LEADER]: Becoming Leader. State: Replica: fbc61ef52a5e4e9ca5740a9db793a40d, State: Running, Role: LEADER
I20260621 02:44:46.713438 15323 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc61ef52a5e4e9ca5740a9db793a40d" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37825 } }
I20260621 02:44:46.713603 15318 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:44:46.716238 15325 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbc61ef52a5e4e9ca5740a9db793a40d. Latest consensus state: current_term: 1 leader_uuid: "fbc61ef52a5e4e9ca5740a9db793a40d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc61ef52a5e4e9ca5740a9db793a40d" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37825 } } }
I20260621 02:44:46.716640 15325 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:46.716332 15324 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbc61ef52a5e4e9ca5740a9db793a40d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc61ef52a5e4e9ca5740a9db793a40d" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37825 } } }
I20260621 02:44:46.716765 15324 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:46.717639 15332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:44:46.723269 15332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:44:46.731357 15332 catalog_manager.cc:1347] Generated new cluster ID: 13dfb42233fd4b33af077e64fe0509cc
I20260621 02:44:46.731521 15332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:44:46.756016 15332 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:44:46.757797 15332 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:44:46.768836 15332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbc61ef52a5e4e9ca5740a9db793a40d: Generated new TSK 0
I20260621 02:44:46.769616 15332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:44:46.792596 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:46.784318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41158 (local address 127.9.166.62:37825)
0621 02:44:46.784502 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:46.784508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:46.784939 (+ 431us) server_negotiation.cc:407] Connection header received
0621 02:44:46.785132 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:46.785138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:46.785222 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:46.785389 (+ 167us) server_negotiation.cc:226] Negotiated authn=JWT
0621 02:44:46.785788 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:46.786939 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:46.788355 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:46.789073 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:46.789199 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:46.789703 (+ 504us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 02:44:46.791971 (+ 2268us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39207/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0621 02:44:46.792278 (+ 307us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41158: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39207/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":58}
W20260621 02:44:46.792721 15321 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41158: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39207/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260621 02:44:46.794399 15347 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:44:46.784120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37825 (local address 127.0.0.1:41158)
0621 02:44:46.784847 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:46.784862 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:46.784980 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:46.785387 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:46.785390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:46.785399 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0621 02:44:46.785641 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:46.785648 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:46.787070 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:46.787073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:46.788227 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:46.788238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:46.789284 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:46.789297 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:46.789426 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:46.789458 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 02:44:46.792164 (+ 2706us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39207/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0621 02:44:46.793976 (+ 1812us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.166.62:37825: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39207/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":1531,"thread_start_us":400,"threads_started":1}
I20260621 02:44:46.795728 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 15252
I20260621 02:44:46.796885 15313 generic_service.cc:196] Checking for leaks (request via RPC)
E20260621 02:44:46.804850 15348 webserver.cc:533] Webserver: sslize failed:
E20260621 02:44:46.804970 15348 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260621 02:44:46.888981 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 15252
2026-06-21T02:44:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (703 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-21T02:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:44:46Z Disabled control of system clock
I20260621 02:44:46.967957 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:33551
--webserver_interface=127.9.166.62
--webserver_port=0
--builtin_ntp_servers=127.9.166.20:43651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:33551
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009844354615-9880-0/testchainca.cert with env {}
W20260621 02:44:47.280493 15353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:47.280781 15353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:47.280933 15353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:47.288009 15353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:44:47.288141 15353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:47.288230 15353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:47.288336 15353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:44:47.288420 15353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260621 02:44:47.288508 15353 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260621 02:44:47.288563 15353 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260621 02:44:47.298071 15353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:33551
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:33551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009844354615-9880-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:47.299248 15353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:47.300505 15353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:47.311712 15358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:47.311765 15359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:47.311827 15361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:47.312702 15353 server_base.cc:1034] running on GCE node
I20260621 02:44:47.313460 15353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:47.314756 15353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:47.315974 15353 hybrid_clock.cc:648] HybridClock initialized: now 1782009887315943 us; error 45 us; skew 500 ppm
I20260621 02:44:47.316309 15353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:44:47.318605 15353 webserver.cc:460] Webserver started at http://127.9.166.62:43843/ using document root <none> and password file <none>
I20260621 02:44:47.319271 15353 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:47.319394 15353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:47.319728 15353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:47.322506 15353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "47f45378256a4af4bf92032945245596"
format_stamp: "Formatted at 2026-06-21 02:44:47 on dist-test-slave-hvgn"
server_key: "d7e7fa4d4484d6bb00edfa2127deb7b6"
server_key_iv: "b25671f02f115d5b435b4b732d926d08"
server_key_version: "encryptionkey@0"
I20260621 02:44:47.323455 15353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "47f45378256a4af4bf92032945245596"
format_stamp: "Formatted at 2026-06-21 02:44:47 on dist-test-slave-hvgn"
server_key: "d7e7fa4d4484d6bb00edfa2127deb7b6"
server_key_iv: "b25671f02f115d5b435b4b732d926d08"
server_key_version: "encryptionkey@0"
I20260621 02:44:47.328334 15353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260621 02:44:47.331635 15367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:47.332926 15353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:44:47.333108 15353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "47f45378256a4af4bf92032945245596"
format_stamp: "Formatted at 2026-06-21 02:44:47 on dist-test-slave-hvgn"
server_key: "d7e7fa4d4484d6bb00edfa2127deb7b6"
server_key_iv: "b25671f02f115d5b435b4b732d926d08"
server_key_version: "encryptionkey@0"
I20260621 02:44:47.333330 15353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:47.350953 15353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:47.351807 15353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:47.352077 15353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:47.370030 15353 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:33551
I20260621 02:44:47.370048 15418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:33551 every 8 connection(s)
I20260621 02:44:47.371433 15353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:44:47.373014 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 15353
I20260621 02:44:47.373206 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:44:47.373823 9880 external_mini_cluster.cc:1428] Setting key fdcdd0676eaefc912ac7d00b0df49d9c
I20260621 02:44:47.375620 15419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:47.389020 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:47.375788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47918 (local address 127.9.166.62:33551)
0621 02:44:47.376788 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:47.376805 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:47.376856 (+ 51us) server_negotiation.cc:407] Connection header received
0621 02:44:47.378837 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:47.378864 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:47.379485 (+ 621us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:47.379964 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:47.380487 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:47.382521 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:47.383816 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:47.384803 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:47.384979 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:47.386060 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:47.386070 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:47.386091 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:44:47.386954 (+ 863us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:47.387045 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:44:47.387077 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:47.387250 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:47.387459 (+ 209us) server_negotiation.cc:299] Negotiation successful
0621 02:44:47.387802 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":294,"threads_started":1}
I20260621 02:44:47.390157 15419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596: Bootstrap starting.
I20260621 02:44:47.392899 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:44:47.393923 15419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:47.395146 15419 log.cc:824] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:47.398204 15419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596: No bootstrap required, opened a new log
I20260621 02:44:47.405462 15419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f45378256a4af4bf92032945245596" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 33551 } }
I20260621 02:44:47.405853 15419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:47.405983 15419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f45378256a4af4bf92032945245596, State: Initialized, Role: FOLLOWER
I20260621 02:44:47.406529 15419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f45378256a4af4bf92032945245596" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 33551 } }
I20260621 02:44:47.406759 15419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:44:47.406901 15419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:44:47.407074 15419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:47.409124 15419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f45378256a4af4bf92032945245596" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 33551 } }
I20260621 02:44:47.409595 15419 leader_election.cc:302] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47f45378256a4af4bf92032945245596; no voters:
I20260621 02:44:47.410135 15419 leader_election.cc:288] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:44:47.410275 15424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:47.410964 15424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596 [term 1 LEADER]: Becoming Leader. State: Replica: 47f45378256a4af4bf92032945245596, State: Running, Role: LEADER
I20260621 02:44:47.411564 15424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f45378256a4af4bf92032945245596" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 33551 } }
I20260621 02:44:47.412263 15419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:44:47.414307 15426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47f45378256a4af4bf92032945245596. Latest consensus state: current_term: 1 leader_uuid: "47f45378256a4af4bf92032945245596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f45378256a4af4bf92032945245596" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 33551 } } }
I20260621 02:44:47.414682 15426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:47.414566 15425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47f45378256a4af4bf92032945245596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f45378256a4af4bf92032945245596" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 33551 } } }
I20260621 02:44:47.414840 15425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596 [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:47.416535 15432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:44:47.421038 15432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:44:47.428853 15432 catalog_manager.cc:1347] Generated new cluster ID: ace8f1368d2e4a719e42909588edb9e8
I20260621 02:44:47.428962 15432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:44:47.458261 15432 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:44:47.459601 15432 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:44:47.466642 15432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47f45378256a4af4bf92032945245596: Generated new TSK 0
I20260621 02:44:47.467361 15432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:44:48.392800 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:47.488480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47930 (local address 127.9.166.62:33551)
0621 02:44:47.488665 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:47.488671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:47.489071 (+ 400us) server_negotiation.cc:407] Connection header received
0621 02:44:47.489289 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:47.489295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:47.489379 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:47.489536 (+ 157us) server_negotiation.cc:226] Negotiated authn=JWT
0621 02:44:47.489938 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:47.491041 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:47.492425 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:47.493129 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:47.493244 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:47.493683 (+ 439us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0621 02:44:48.392227 (+898544us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0621 02:44:48.392590 (+ 363us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47930: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":60}
W20260621 02:44:48.392939 15422 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47930: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260621 02:44:48.393101 15448 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:44:47.488345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:33551 (local address 127.0.0.1:47930)
0621 02:44:47.488981 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:47.488996 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:47.489103 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:47.489539 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:47.489542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:47.489550 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0621 02:44:47.489802 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:47.489811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:47.491173 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:47.491177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:47.492282 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:47.492293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:47.493290 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:47.493294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:47.493424 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:47.493455 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0621 02:44:48.392527 (+899072us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0621 02:44:48.392767 (+ 240us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.166.62:33551: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":347,"threads_started":1}
I20260621 02:44:48.394507 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 15353
I20260621 02:44:48.395653 15414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:48.496980 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 15353
2026-06-21T02:44:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1590 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (404 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (408 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:44:49 dist-test-slave-hvgn krb5kdc[15463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:44:49 dist-test-slave-hvgn krb5kdc[15463](info): set up 2 sockets
Jun 21 02:44:49 dist-test-slave-hvgn krb5kdc[15463](info): commencing operation
krb5kdc: starting...
W20260621 02:44:50.387867 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:44:50 dist-test-slave-hvgn krb5kdc[15463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:44:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:44:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:44:50.580721 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:42149
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:42149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:50.893761 15476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:50.894075 15476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:50.894246 15476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:50.901249 15476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:44:50.901387 15476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:50.901494 15476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:50.901597 15476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:44:50.901678 15476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:44:50.911036 15476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:42149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:42149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:50.912323 15476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:50.913615 15476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:50.924710 15482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:50.924777 15481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:50.924759 15484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:50.925853 15476 server_base.cc:1034] running on GCE node
I20260621 02:44:50.926577 15476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:50.927909 15476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:50.929123 15476 hybrid_clock.cc:648] HybridClock initialized: now 1782009890929067 us; error 33 us; skew 500 ppm
I20260621 02:44:50.929443 15476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:50 dist-test-slave-hvgn krb5kdc[15463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009890, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:50.935003 15476 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:44:50.936519 15476 webserver.cc:460] Webserver started at http://127.9.166.62:42487/ using document root <none> and password file <none>
I20260621 02:44:50.937179 15476 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:50.937305 15476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:50.937623 15476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:50.940553 15476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "90811deaf00c4359af992e7d71f533ee"
format_stamp: "Formatted at 2026-06-21 02:44:50 on dist-test-slave-hvgn"
server_key: "efe985dd2ce42607a8e12f68517a30cd"
server_key_iv: "71e71c4aad46a3bc7b3dd6d01e8c778f"
server_key_version: "encryptionkey@0"
I20260621 02:44:50.941308 15476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "90811deaf00c4359af992e7d71f533ee"
format_stamp: "Formatted at 2026-06-21 02:44:50 on dist-test-slave-hvgn"
server_key: "efe985dd2ce42607a8e12f68517a30cd"
server_key_iv: "71e71c4aad46a3bc7b3dd6d01e8c778f"
server_key_version: "encryptionkey@0"
I20260621 02:44:50.946179 15476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260621 02:44:50.949491 15491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:50.950744 15476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 02:44:50.950907 15476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "90811deaf00c4359af992e7d71f533ee"
format_stamp: "Formatted at 2026-06-21 02:44:50 on dist-test-slave-hvgn"
server_key: "efe985dd2ce42607a8e12f68517a30cd"
server_key_iv: "71e71c4aad46a3bc7b3dd6d01e8c778f"
server_key_version: "encryptionkey@0"
I20260621 02:44:50.951071 15476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:50.967275 15476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:50.971163 15476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:50.971478 15476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:50.989390 15476 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:42149
I20260621 02:44:50.989429 15542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:42149 every 8 connection(s)
I20260621 02:44:50.990743 15476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:44:50.994444 15543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:50.998078 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 15476
I20260621 02:44:50.998288 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:44:50.999109 9880 external_mini_cluster.cc:1428] Setting key c5c3aff706ce0c2d82cb05427b501ae7
I20260621 02:44:51.007946 15543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee: Bootstrap starting.
I20260621 02:44:51.011701 15543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:51.012981 15543 log.cc:824] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee: Log is configured to *not* fsync() on all Append() calls
Jun 21 02:44:51 dist-test-slave-hvgn krb5kdc[15463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:51.015702 15543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee: No bootstrap required, opened a new log
I20260621 02:44:51.021569 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:51.001198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35844 (local address 127.9.166.62:42149)
0621 02:44:51.001971 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:51.001982 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:51.002022 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:44:51.003765 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:51.003788 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:51.004185 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:51.004495 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:51.006215 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:51.007544 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:51.009223 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:51.009870 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:51.009998 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:51.014509 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:51.014516 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:51.014529 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:51.014561 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:51.017715 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:51.018398 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:51.018406 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:51.018423 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:51.018575 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:51.018948 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:51.018953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:51.018957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:51.019487 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:51.019810 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:51.020223 (+ 413us) server_negotiation.cc:299] Negotiation successful
0621 02:44:51.020606 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":383,"threads_started":1}
I20260621 02:44:51.023331 15543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90811deaf00c4359af992e7d71f533ee" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 42149 } }
I20260621 02:44:51.023761 15543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:51.023877 15543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90811deaf00c4359af992e7d71f533ee, State: Initialized, Role: FOLLOWER
I20260621 02:44:51.024499 15543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90811deaf00c4359af992e7d71f533ee" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 42149 } }
I20260621 02:44:51.024748 15543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:44:51.024888 15543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:44:51.025059 15543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:51.027076 15543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90811deaf00c4359af992e7d71f533ee" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 42149 } }
I20260621 02:44:51.027638 15543 leader_election.cc:302] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90811deaf00c4359af992e7d71f533ee; no voters:
I20260621 02:44:51.028151 15543 leader_election.cc:288] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:44:51.028282 15548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:51.028967 15548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee [term 1 LEADER]: Becoming Leader. State: Replica: 90811deaf00c4359af992e7d71f533ee, State: Running, Role: LEADER
I20260621 02:44:51.029426 15548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90811deaf00c4359af992e7d71f533ee" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 42149 } }
I20260621 02:44:51.030280 15543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:44:51.032491 15550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90811deaf00c4359af992e7d71f533ee. Latest consensus state: current_term: 1 leader_uuid: "90811deaf00c4359af992e7d71f533ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90811deaf00c4359af992e7d71f533ee" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 42149 } } }
I20260621 02:44:51.032858 15550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:51.032729 15549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90811deaf00c4359af992e7d71f533ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90811deaf00c4359af992e7d71f533ee" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 42149 } } }
I20260621 02:44:51.033103 15549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:51.035082 15556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:44:51.040418 15556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:44:51.049178 15556 catalog_manager.cc:1347] Generated new cluster ID: 4f71e22292ca40f3a42ee05fc63c7691
I20260621 02:44:51.049314 15556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:44:51.086066 15556 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:44:51.087486 15556 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:44:51.098537 15556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90811deaf00c4359af992e7d71f533ee: Generated new TSK 0
I20260621 02:44:51.099834 15556 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
I20260621 02:44:51.154425 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:42149
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:51.472218 15569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:51.472483 15569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:51.472638 15569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:51.479856 15569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:51.479955 15569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:51.480181 15569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:44:51.489454 15569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:42149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:51.490677 15569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:51.492009 15569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:51.505250 15574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:51.505210 15577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:51.505298 15575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:51.506096 15569 server_base.cc:1034] running on GCE node
I20260621 02:44:51.506722 15569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:51.507694 15569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:51.508911 15569 hybrid_clock.cc:648] HybridClock initialized: now 1782009891508867 us; error 35 us; skew 500 ppm
I20260621 02:44:51.509253 15569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:51 dist-test-slave-hvgn krb5kdc[15463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009891, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:51.512385 15569 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:44:51.513847 15569 webserver.cc:460] Webserver started at http://127.9.166.1:44963/ using document root <none> and password file <none>
I20260621 02:44:51.514518 15569 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:51.514659 15569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:51.514997 15569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:51.517797 15569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "a239cf930e654931a1953999cfd59e02"
format_stamp: "Formatted at 2026-06-21 02:44:51 on dist-test-slave-hvgn"
server_key: "9587790cbbc9906ba7fb6ae069809da5"
server_key_iv: "659c823e8201d9b8c9276a3f278f89b4"
server_key_version: "encryptionkey@0"
I20260621 02:44:51.518556 15569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "a239cf930e654931a1953999cfd59e02"
format_stamp: "Formatted at 2026-06-21 02:44:51 on dist-test-slave-hvgn"
server_key: "9587790cbbc9906ba7fb6ae069809da5"
server_key_iv: "659c823e8201d9b8c9276a3f278f89b4"
server_key_version: "encryptionkey@0"
I20260621 02:44:51.523620 15569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260621 02:44:51.526857 15584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:51.528185 15569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:44:51.528353 15569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "a239cf930e654931a1953999cfd59e02"
format_stamp: "Formatted at 2026-06-21 02:44:51 on dist-test-slave-hvgn"
server_key: "9587790cbbc9906ba7fb6ae069809da5"
server_key_iv: "659c823e8201d9b8c9276a3f278f89b4"
server_key_version: "encryptionkey@0"
I20260621 02:44:51.528544 15569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:51.544406 15569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:51.548950 15569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:51.549250 15569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:51.550209 15569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:51.551554 15569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:51.551653 15569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:51.551800 15569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:51.551873 15569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:51.585034 15569 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:43355
I20260621 02:44:51.585081 15696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:43355 every 8 connection(s)
I20260621 02:44:51.586315 15569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:44:51.591789 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 15569
I20260621 02:44:51.591992 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:44:51.592607 9880 external_mini_cluster.cc:1428] Setting key bfad532691e3ba418dd140ca43aab78f
Jun 21 02:44:51 dist-test-slave-hvgn krb5kdc[15463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009891, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:51.606817 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:51.588888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:60253 (local address 127.9.166.62:42149)
0621 02:44:51.589351 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:51.589355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:51.590274 (+ 919us) server_negotiation.cc:407] Connection header received
0621 02:44:51.592200 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:51.592205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:51.592290 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:51.592413 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:51.594192 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:51.595305 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:51.596859 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:51.597540 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:51.597699 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:51.601328 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:51.601333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:51.601336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:51.601366 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:51.603902 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:51.604594 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:51.604601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:51.604604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:51.604727 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:51.605171 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:51.605175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:51.605179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:51.605431 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:51.605574 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:51.606233 (+ 659us) server_negotiation.cc:299] Negotiation successful
0621 02:44:51.606450 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":257,"threads_started":1}
I20260621 02:44:51.607865 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:51.589408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:42149 (local address 127.9.166.1:60253)
0621 02:44:51.590099 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:51.590135 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:51.591898 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:51.592658 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:51.592666 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:51.593283 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:51.594018 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:51.594041 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:51.595505 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:51.595509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:51.596718 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:51.596730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:51.597789 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:51.597793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:51.598047 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:51.598796 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:51.598818 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:51.601097 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:51.604046 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:51.604054 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:51.604071 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:51.604474 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:51.604854 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:51.604858 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:51.604862 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:51.605045 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:51.605562 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:51.605569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:51.605890 (+ 321us) client_negotiation.cc:769] Sending connection context
0621 02:44:51.606166 (+ 276us) client_negotiation.cc:240] Negotiation successful
0621 02:44:51.606672 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":285,"threads_started":1}
I20260621 02:44:51.609506 15697 heartbeater.cc:347] Connected to a master server at 127.9.166.62:42149
I20260621 02:44:51.609815 15697 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:51.610519 15697 heartbeater.cc:511] Master 127.9.166.62:42149 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.166.2@KRBTEST.COM; defaulting to no policy
I20260621 02:44:51.612571 15508 ts_manager.cc:194] Registered new tserver with Master: a239cf930e654931a1953999cfd59e02 (127.9.166.1:43355)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.2@KRBTEST.COM" created.
I20260621 02:44:51.614297 15508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:60253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
I20260621 02:44:51.636411 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:42149
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:51.948269 15703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:51.948565 15703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:51.948731 15703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:51.955861 15703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:51.955998 15703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:51.956167 15703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:44:51.965365 15703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:42149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:51.966591 15703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:51.967844 15703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:51.980414 15708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:51.980456 15709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:51.980371 15711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:51.981294 15703 server_base.cc:1034] running on GCE node
I20260621 02:44:51.981850 15703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:51.982702 15703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:51.983888 15703 hybrid_clock.cc:648] HybridClock initialized: now 1782009891983815 us; error 62 us; skew 500 ppm
I20260621 02:44:51.984179 15703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:51 dist-test-slave-hvgn krb5kdc[15463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009891, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:51.987099 15703 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:44:51.988456 15703 webserver.cc:460] Webserver started at http://127.9.166.2:34967/ using document root <none> and password file <none>
I20260621 02:44:51.989126 15703 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:51.989271 15703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:51.989583 15703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:51.992350 15703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "eef2ed80ad874d479ec1fd7ee0700d8b"
format_stamp: "Formatted at 2026-06-21 02:44:51 on dist-test-slave-hvgn"
server_key: "86aac23c06d714066464242c94b3cb27"
server_key_iv: "27c92572280dd52262ce111a6c53e4f2"
server_key_version: "encryptionkey@0"
I20260621 02:44:51.993117 15703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "eef2ed80ad874d479ec1fd7ee0700d8b"
format_stamp: "Formatted at 2026-06-21 02:44:51 on dist-test-slave-hvgn"
server_key: "86aac23c06d714066464242c94b3cb27"
server_key_iv: "27c92572280dd52262ce111a6c53e4f2"
server_key_version: "encryptionkey@0"
I20260621 02:44:51.998334 15703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260621 02:44:52.001806 15718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:52.003085 15703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 02:44:52.003259 15703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "eef2ed80ad874d479ec1fd7ee0700d8b"
format_stamp: "Formatted at 2026-06-21 02:44:51 on dist-test-slave-hvgn"
server_key: "86aac23c06d714066464242c94b3cb27"
server_key_iv: "27c92572280dd52262ce111a6c53e4f2"
server_key_version: "encryptionkey@0"
I20260621 02:44:52.003539 15703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:52.033771 15703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:52.037982 15703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:52.038269 15703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:52.039186 15703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:52.040460 15703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:52.040530 15703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:52.040709 15703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:52.040759 15703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:52.071559 15703 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:45887
I20260621 02:44:52.071606 15830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:45887 every 8 connection(s)
I20260621 02:44:52.072816 15703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:44:52.074481 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 15703
I20260621 02:44:52.074641 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:44:52.075132 9880 external_mini_cluster.cc:1428] Setting key ac80e8162cfd3e2c4e4e0e06be99e10d
Jun 21 02:44:52 dist-test-slave-hvgn krb5kdc[15463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009891, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:52.093994 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:52.075325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:56509 (local address 127.9.166.62:42149)
0621 02:44:52.075571 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:52.075578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:52.076824 (+ 1246us) server_negotiation.cc:407] Connection header received
0621 02:44:52.079502 (+ 2678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:52.079506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:52.079576 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:52.079658 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:52.081676 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:52.082365 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:52.084598 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:52.085017 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:52.085148 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:52.088477 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:52.088482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:52.088485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:52.088513 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:52.090996 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:52.091832 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:52.091844 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:52.091847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:52.091964 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:52.092370 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:52.092375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:52.092378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:52.092628 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:52.092779 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:52.093429 (+ 650us) server_negotiation.cc:299] Negotiation successful
0621 02:44:52.093645 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
WARNING: no policy specified for kudu/127.9.166.3@KRBTEST.COM; defaulting to no policy
I20260621 02:44:52.094918 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:52.075911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:42149 (local address 127.9.166.2:56509)
0621 02:44:52.076641 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:52.076709 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:52.079185 (+ 2476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:52.079940 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:52.079955 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:52.080704 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:52.081507 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:52.081533 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:52.082515 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:52.082523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:52.084436 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:52.084448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:52.085233 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:52.085242 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:52.085469 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:52.086124 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:52.086145 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:52.088266 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:52.091128 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:52.091136 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:52.091150 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:52.091711 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:52.092070 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:52.092076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:52.092080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:52.092256 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:52.092753 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:52.092761 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:52.093079 (+ 318us) client_negotiation.cc:769] Sending connection context
0621 02:44:52.093338 (+ 259us) client_negotiation.cc:240] Negotiation successful
0621 02:44:52.093832 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":329,"threads_started":1}
I20260621 02:44:52.096391 15831 heartbeater.cc:347] Connected to a master server at 127.9.166.62:42149
I20260621 02:44:52.096737 15831 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.3@KRBTEST.COM" created.
I20260621 02:44:52.097600 15831 heartbeater.cc:511] Master 127.9.166.62:42149 requested a full tablet report, sending...
I20260621 02:44:52.099174 15508 ts_manager.cc:194] Registered new tserver with Master: eef2ed80ad874d479ec1fd7ee0700d8b (127.9.166.2:45887)
I20260621 02:44:52.100397 15508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:56509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
I20260621 02:44:52.119361 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:42149
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:52.428928 15836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:52.429241 15836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:52.429406 15836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:52.437569 15836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:52.437721 15836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:52.437923 15836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:44:52.447552 15836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:42149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:52.448868 15836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:52.450208 15836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:52.463198 15842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:52.463270 15841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:52.463227 15844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:52.468807 15836 server_base.cc:1034] running on GCE node
I20260621 02:44:52.469348 15836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:52.470252 15836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:52.471474 15836 hybrid_clock.cc:648] HybridClock initialized: now 1782009892471440 us; error 33 us; skew 500 ppm
I20260621 02:44:52.471791 15836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:52 dist-test-slave-hvgn krb5kdc[15463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009892, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:52.474857 15836 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:44:52.476405 15836 webserver.cc:460] Webserver started at http://127.9.166.3:39579/ using document root <none> and password file <none>
I20260621 02:44:52.477097 15836 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:52.477238 15836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:52.477548 15836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:52.480405 15836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "e0a2072b63e54f038d1b598937bbeb9d"
format_stamp: "Formatted at 2026-06-21 02:44:52 on dist-test-slave-hvgn"
server_key: "dde57764c9bd94a5cbf66815182a4cc1"
server_key_iv: "c4efbd77f9ca3ccc21e133ff9b00f8b3"
server_key_version: "encryptionkey@0"
I20260621 02:44:52.481174 15836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "e0a2072b63e54f038d1b598937bbeb9d"
format_stamp: "Formatted at 2026-06-21 02:44:52 on dist-test-slave-hvgn"
server_key: "dde57764c9bd94a5cbf66815182a4cc1"
server_key_iv: "c4efbd77f9ca3ccc21e133ff9b00f8b3"
server_key_version: "encryptionkey@0"
I20260621 02:44:52.486068 15836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260621 02:44:52.489656 15851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:52.490978 15836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:44:52.491156 15836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "e0a2072b63e54f038d1b598937bbeb9d"
format_stamp: "Formatted at 2026-06-21 02:44:52 on dist-test-slave-hvgn"
server_key: "dde57764c9bd94a5cbf66815182a4cc1"
server_key_iv: "c4efbd77f9ca3ccc21e133ff9b00f8b3"
server_key_version: "encryptionkey@0"
I20260621 02:44:52.491360 15836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:52.507897 15836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:52.511745 15836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:52.512008 15836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:52.512880 15836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:52.514094 15836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:52.514199 15836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:52.514343 15836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:52.514410 15836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:52.547108 15836 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:34905
I20260621 02:44:52.547148 15963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:34905 every 8 connection(s)
I20260621 02:44:52.548533 15836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:44:52.556566 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 15836
I20260621 02:44:52.556730 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:44:52.557173 9880 external_mini_cluster.cc:1428] Setting key f7cf5d4ee397be8fe1dc423f320066eb
Jun 21 02:44:52 dist-test-slave-hvgn krb5kdc[15463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009892, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:44:52.567308 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:52.550995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:50293 (local address 127.9.166.62:42149)
0621 02:44:52.551162 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:52.551167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:52.552331 (+ 1164us) server_negotiation.cc:407] Connection header received
0621 02:44:52.554100 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:52.554105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:52.554189 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:52.554300 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:52.555870 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:52.556618 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:52.558325 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:52.558734 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:52.558861 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:52.562132 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:52.562137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:52.562140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:52.562171 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:52.564513 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:52.565179 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:52.565189 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:52.565192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:52.565309 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:52.565725 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:52.565733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:52.565736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:52.565983 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:52.566122 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:52.566743 (+ 621us) server_negotiation.cc:299] Negotiation successful
0621 02:44:52.566965 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:44:52.568183 15966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:52.551507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:42149 (local address 127.9.166.3:50293)
0621 02:44:52.552183 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:52.552230 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:52.553833 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:52.554548 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:52.554556 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:52.555112 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:52.555714 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:52.555732 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:52.556840 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:52.556845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:52.558163 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:52.558178 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:52.558940 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:52.558949 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:52.559202 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:52.559863 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:52.559883 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:52.561901 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:52.564642 (+ 2741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:52.564650 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:52.564664 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:52.565055 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:52.565413 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:52.565418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:52.565421 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:52.565605 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:52.566109 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:52.566116 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:52.566409 (+ 293us) client_negotiation.cc:769] Sending connection context
0621 02:44:52.566647 (+ 238us) client_negotiation.cc:240] Negotiation successful
0621 02:44:52.567112 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":269,"threads_started":1}
I20260621 02:44:52.569463 15964 heartbeater.cc:347] Connected to a master server at 127.9.166.62:42149
I20260621 02:44:52.569763 15964 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:52.570466 15964 heartbeater.cc:511] Master 127.9.166.62:42149 requested a full tablet report, sending...
I20260621 02:44:52.572083 15508 ts_manager.cc:194] Registered new tserver with Master: e0a2072b63e54f038d1b598937bbeb9d (127.9.166.3:34905)
I20260621 02:44:52.572826 15508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:50293
I20260621 02:44:52.584071 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/test-admin.keytab.
Jun 21 02:44:52 dist-test-slave-hvgn krb5kdc[15463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:52.612259 9880 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
I20260621 02:44:52.617149 15697 heartbeater.cc:503] Master 127.9.166.62:42149 was elected leader, sending a full tablet report...
W20260621 02:44:52.617512 15971 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:44:52.614846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.9.166.1:43355)
0621 02:44:52.615623 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:52.615630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:52.615658 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:44:52.615720 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:52.615727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:52.615939 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:52.616081 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:52.616894 (+ 813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48558: BlockingRecv error: recv got EOF from 127.0.0.1:48558 (error 108)
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":398,"threads_started":1}
W20260621 02:44:52.617543 9880 init.cc:244] Error while opening credential cache '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/krb5cc)
Jun 21 02:44:52 dist-test-slave-hvgn krb5kdc[15463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:52.619529 9880 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 02:44:52 dist-test-slave-hvgn krb5kdc[15463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:52.620919 9880 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 02:44:52 dist-test-slave-hvgn krb5kdc[15463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:44:52.635659 15971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:52.622428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48566 (local address 127.9.166.1:43355)
0621 02:44:52.622616 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:52.622622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:52.622892 (+ 270us) server_negotiation.cc:407] Connection header received
0621 02:44:52.623028 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:52.623031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:52.623081 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:52.623155 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:52.624095 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:52.625059 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:52.626193 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:52.626724 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:52.626815 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:52.630086 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:52.630095 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:52.630108 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:52.630134 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:52.633123 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:52.633757 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:52.633763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:52.633766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:52.633930 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:52.634252 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:52.634255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:52.634258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:52.634583 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:52.634786 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:52.635099 (+ 313us) server_negotiation.cc:299] Negotiation successful
0621 02:44:52.635377 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260621 02:44:52.640877 15971 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:44:52.639186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48580 (local address 127.9.166.1:43355)
0621 02:44:52.639375 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:52.639383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:52.639648 (+ 265us) server_negotiation.cc:407] Connection header received
0621 02:44:52.639818 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:52.639827 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:52.639887 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:52.639980 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:52.640632 (+ 652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48580: BlockingRecv error: recv got EOF from 127.0.0.1:48580 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260621 02:44:52.641040 9880 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/krb5cc' of type FILE
Jun 21 02:44:52 dist-test-slave-hvgn krb5kdc[15463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:52.642422 9880 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 02:44:52 dist-test-slave-hvgn krb5kdc[15463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:52.643769 9880 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 02:44:52 dist-test-slave-hvgn krb5kdc[15463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:44:52.657034 15971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:52.645007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48588 (local address 127.9.166.1:43355)
0621 02:44:52.645168 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:52.645174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:52.645319 (+ 145us) server_negotiation.cc:407] Connection header received
0621 02:44:52.645424 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:52.645428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:52.645476 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:52.645550 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:52.646597 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:52.647296 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:52.648495 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:52.648914 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:52.649052 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:52.652173 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:52.652183 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:52.652186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:52.652215 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:52.654577 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:52.655234 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:52.655240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:52.655248 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:52.655361 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:52.655742 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:52.655747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:52.655750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:52.656002 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:52.656246 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:52.656527 (+ 281us) server_negotiation.cc:299] Negotiation successful
0621 02:44:52.656727 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260621 02:44:52.660717 15971 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:44:52.659343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48598 (local address 127.9.166.1:43355)
0621 02:44:52.659533 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:52.659538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:52.659712 (+ 174us) server_negotiation.cc:407] Connection header received
0621 02:44:52.659811 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:52.659814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:52.659874 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:52.659947 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:52.660524 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48598: BlockingRecv error: recv got EOF from 127.0.0.1:48598 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260621 02:44:52.660852 9880 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1782009844354615-9880-0/krb5kdc/krb5cc' of type FILE
Jun 21 02:44:52 dist-test-slave-hvgn krb5kdc[15463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:52.662256 9880 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 21 02:44:52 dist-test-slave-hvgn krb5kdc[15463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:52.663549 9880 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 21 02:44:52 dist-test-slave-hvgn krb5kdc[15463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:44:52.676818 15971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:52.664644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48604 (local address 127.9.166.1:43355)
0621 02:44:52.664821 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:52.664826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:52.665002 (+ 176us) server_negotiation.cc:407] Connection header received
0621 02:44:52.665122 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:52.665125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:52.665174 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:52.665261 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:52.666264 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:52.666940 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:52.668237 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:52.668650 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:52.668737 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:52.672286 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:52.672290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:52.672293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:52.672321 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:52.674592 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:52.675155 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:52.675159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:52.675161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:52.675286 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:52.675677 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:52.675680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:52.675682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:52.675946 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:52.676163 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:52.676360 (+ 197us) server_negotiation.cc:299] Negotiation successful
0621 02:44:52.676507 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:44:52.678156 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 15569
I20260621 02:44:52.689225 15971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:52.678763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48616 (local address 127.9.166.1:43355)
0621 02:44:52.678889 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:52.678893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:52.679257 (+ 364us) server_negotiation.cc:407] Connection header received
0621 02:44:52.679426 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:52.679431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:52.679495 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:52.679563 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:52.680502 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:52.681153 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:52.682283 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:52.682659 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:52.682765 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:52.684623 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:52.684627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:52.684630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:52.684654 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:52.686792 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:52.687558 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:52.687566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:52.687571 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:52.687787 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:52.688209 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:52.688214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:52.688216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:52.688454 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:52.688642 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:52.688843 (+ 201us) server_negotiation.cc:299] Negotiation successful
0621 02:44:52.688978 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 02:44:52.689752 15692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:52.772874 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 15569
I20260621 02:44:52.788686 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 15703
Jun 21 02:44:52 dist-test-slave-hvgn krb5kdc[15463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.2@KRBTEST.COM
I20260621 02:44:52.803653 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:52.789545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.9.166.2:45887)
0621 02:44:52.790023 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:52.790030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:52.790054 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:44:52.790113 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:52.790119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:52.790336 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:52.790498 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:52.791477 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:52.792443 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:52.793646 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:52.794267 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:52.794401 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:52.798256 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:52.798272 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:52.798286 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:52.798318 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:52.800945 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:52.801595 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:52.801603 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:52.801605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:52.801720 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:52.802098 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:52.802104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:52.802107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:52.802430 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:52.802729 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:52.803034 (+ 305us) server_negotiation.cc:299] Negotiation successful
0621 02:44:52.803289 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":229,"threads_started":1}
I20260621 02:44:52.804834 15826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:52.885682 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 15703
I20260621 02:44:52.901285 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 15836
Jun 21 02:44:52 dist-test-slave-hvgn krb5kdc[15463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.3@KRBTEST.COM
I20260621 02:44:52.916285 15986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:52.902283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.9.166.3:34905)
0621 02:44:52.902925 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:52.902932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:52.902959 (+ 27us) server_negotiation.cc:407] Connection header received
0621 02:44:52.903021 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:52.903027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:52.903234 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:52.903361 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:52.904415 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:52.905408 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:52.906574 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:52.907112 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:52.907203 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:52.911063 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:52.911073 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:52.911087 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:52.911118 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:52.913845 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:52.914449 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:52.914456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:52.914458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:52.914570 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:52.914947 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:52.914950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:52.914952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:52.915270 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:52.915572 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:52.915830 (+ 258us) server_negotiation.cc:299] Negotiation successful
0621 02:44:52.916010 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":369,"threads_started":1}
I20260621 02:44:52.917547 15959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:53.000586 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 15836
I20260621 02:44:53.016542 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 15476
I20260621 02:44:53.017645 15538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:53.118588 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 15476
2026-06-21T02:44:53Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3811 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:44:53 dist-test-slave-hvgn krb5kdc[15991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:44:53 dist-test-slave-hvgn krb5kdc[15991](info): set up 2 sockets
Jun 21 02:44:53 dist-test-slave-hvgn krb5kdc[15991](info): commencing operation
krb5kdc: starting...
W20260621 02:44:54.204083 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.006s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:44:54 dist-test-slave-hvgn krb5kdc[15991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:44:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:44:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.62.keytab.
Entry for principal oryx/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.62.keytab.
I20260621 02:44:54.394217 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:37723
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.62.keytab
--principal=oryx/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:37723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:54.704838 16004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:54.705094 16004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:54.705194 16004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:54.712070 16004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:44:54.712164 16004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:54.712221 16004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:54.712275 16004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:44:54.712358 16004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:44:54.721617 16004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:37723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.62.keytab
--principal=oryx/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:37723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:54.722791 16004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:54.724066 16004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:54.735090 16010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:54.735188 16009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:54.735221 16012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:54.735987 16004 server_base.cc:1034] running on GCE node
I20260621 02:44:54.736740 16004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:54.738137 16004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:54.739428 16004 hybrid_clock.cc:648] HybridClock initialized: now 1782009894739356 us; error 51 us; skew 500 ppm
I20260621 02:44:54.739786 16004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:54 dist-test-slave-hvgn krb5kdc[15991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009894, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:54.742815 16004 init.cc:377] Logged in from keytab as oryx/127.9.166.62@KRBTEST.COM (short username oryx)
I20260621 02:44:54.744429 16004 webserver.cc:460] Webserver started at http://127.9.166.62:34429/ using document root <none> and password file <none>
I20260621 02:44:54.745102 16004 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:54.745275 16004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:54.745574 16004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:54.749342 16004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "2296b5aeb8f34f529d7886dae3a889cf"
format_stamp: "Formatted at 2026-06-21 02:44:54 on dist-test-slave-hvgn"
server_key: "37b764e576ff71f335285659563552cf"
server_key_iv: "939ae3b57d96db2cadb8858b045b59b7"
server_key_version: "encryptionkey@0"
I20260621 02:44:54.750166 16004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "2296b5aeb8f34f529d7886dae3a889cf"
format_stamp: "Formatted at 2026-06-21 02:44:54 on dist-test-slave-hvgn"
server_key: "37b764e576ff71f335285659563552cf"
server_key_iv: "939ae3b57d96db2cadb8858b045b59b7"
server_key_version: "encryptionkey@0"
I20260621 02:44:54.755223 16004 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 02:44:54.758597 16019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:54.759946 16004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:44:54.760125 16004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "2296b5aeb8f34f529d7886dae3a889cf"
format_stamp: "Formatted at 2026-06-21 02:44:54 on dist-test-slave-hvgn"
server_key: "37b764e576ff71f335285659563552cf"
server_key_iv: "939ae3b57d96db2cadb8858b045b59b7"
server_key_version: "encryptionkey@0"
I20260621 02:44:54.760308 16004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:54.781625 16004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:54.785804 16004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:54.786091 16004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:54.804147 16004 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:37723
I20260621 02:44:54.804183 16070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:37723 every 8 connection(s)
I20260621 02:44:54.805442 16004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:44:54.809017 16071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:54.810842 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 16004
I20260621 02:44:54.811051 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:44:54.811895 9880 external_mini_cluster.cc:1428] Setting key 1d9d4ecf5cd55bd91f027c737c1f78e5
I20260621 02:44:54.820559 16071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf: Bootstrap starting.
I20260621 02:44:54.823493 16071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:54.824431 16071 log.cc:824] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf: Log is configured to *not* fsync() on all Append() calls
Jun 21 02:44:54 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM
I20260621 02:44:54.827530 16071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf: No bootstrap required, opened a new log
I20260621 02:44:54.833348 16074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:54.814140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42354 (local address 127.9.166.62:37723)
0621 02:44:54.814860 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:54.814871 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:54.814913 (+ 42us) server_negotiation.cc:407] Connection header received
0621 02:44:54.816556 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:54.816577 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:54.816934 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:54.817302 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:54.818718 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:54.820044 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:54.821332 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:54.822005 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:54.822193 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:54.826368 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:54.826376 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:54.826389 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:54.826424 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:54.829498 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:54.830268 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:54.830273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:54.830279 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:54.830436 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:54.830835 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:54.830839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:54.830840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:54.831311 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:54.831628 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:54.831981 (+ 353us) server_negotiation.cc:299] Negotiation successful
0621 02:44:54.832339 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":315,"threads_started":1}
I20260621 02:44:54.836591 16071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } }
I20260621 02:44:54.836930 16071 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:54.837028 16071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2296b5aeb8f34f529d7886dae3a889cf, State: Initialized, Role: FOLLOWER
I20260621 02:44:54.837530 16071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } }
I20260621 02:44:54.837756 16071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:44:54.837941 16071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:44:54.838111 16071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:54.840219 16071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } }
I20260621 02:44:54.840690 16071 leader_election.cc:302] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2296b5aeb8f34f529d7886dae3a889cf; no voters:
I20260621 02:44:54.841234 16071 leader_election.cc:288] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:44:54.841384 16076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:54.842576 16076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [term 1 LEADER]: Becoming Leader. State: Replica: 2296b5aeb8f34f529d7886dae3a889cf, State: Running, Role: LEADER
I20260621 02:44:54.843151 16076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } }
I20260621 02:44:54.843282 16071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:44:54.846378 16078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2296b5aeb8f34f529d7886dae3a889cf. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } }
I20260621 02:44:54.846490 16077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } }
I20260621 02:44:54.846782 16078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:54.846877 16077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:54.849658 16086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:44:54.854586 16086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:44:54.862629 16086 catalog_manager.cc:1347] Generated new cluster ID: c80c61a5840d48a68a453a9bbfc9acb9
I20260621 02:44:54.862746 16086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:44:54.905268 16086 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:44:54.906551 16086 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:44:54.915601 16086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf: Generated new TSK 0
I20260621 02:44:54.916368 16086 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.1.keytab.
Entry for principal oryx/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.1.keytab.
I20260621 02:44:54.964376 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:37723
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.1.keytab
--principal=oryx/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:55.281458 16097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:55.281714 16097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:55.281807 16097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:55.288779 16097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:55.288874 16097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:55.288988 16097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:44:55.298203 16097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.1.keytab
--principal=oryx/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:37723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:55.299353 16097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:55.300609 16097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:55.313071 16103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:55.313169 16102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:55.313150 16105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:55.313849 16097 server_base.cc:1034] running on GCE node
I20260621 02:44:55.314406 16097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:55.315261 16097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:55.316490 16097 hybrid_clock.cc:648] HybridClock initialized: now 1782009895316396 us; error 65 us; skew 500 ppm
I20260621 02:44:55.316782 16097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:55 dist-test-slave-hvgn krb5kdc[15991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009895, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:55.319733 16097 init.cc:377] Logged in from keytab as oryx/127.9.166.1@KRBTEST.COM (short username oryx)
I20260621 02:44:55.320969 16097 webserver.cc:460] Webserver started at http://127.9.166.1:46035/ using document root <none> and password file <none>
I20260621 02:44:55.321604 16097 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:55.321700 16097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:55.321947 16097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:55.324681 16097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "df8a57d499a54ac5bc5d7a5579e910cf"
format_stamp: "Formatted at 2026-06-21 02:44:55 on dist-test-slave-hvgn"
server_key: "b95a64220cb85e192b0337eccb03b309"
server_key_iv: "08d9a441677b0922df4bd2de63f1c573"
server_key_version: "encryptionkey@0"
I20260621 02:44:55.325328 16097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "df8a57d499a54ac5bc5d7a5579e910cf"
format_stamp: "Formatted at 2026-06-21 02:44:55 on dist-test-slave-hvgn"
server_key: "b95a64220cb85e192b0337eccb03b309"
server_key_iv: "08d9a441677b0922df4bd2de63f1c573"
server_key_version: "encryptionkey@0"
I20260621 02:44:55.330000 16097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260621 02:44:55.333279 16112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:55.334645 16097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:44:55.334774 16097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "df8a57d499a54ac5bc5d7a5579e910cf"
format_stamp: "Formatted at 2026-06-21 02:44:55 on dist-test-slave-hvgn"
server_key: "b95a64220cb85e192b0337eccb03b309"
server_key_iv: "08d9a441677b0922df4bd2de63f1c573"
server_key_version: "encryptionkey@0"
I20260621 02:44:55.334966 16097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:55.350457 16097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:55.354133 16097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:55.354400 16097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:55.355273 16097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:55.356524 16097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:55.356601 16097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:55.356756 16097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:55.356813 16097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:55.390946 16097 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:35761
I20260621 02:44:55.390990 16224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:35761 every 8 connection(s)
I20260621 02:44:55.392360 16097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:44:55.402558 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 16097
I20260621 02:44:55.402804 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:44:55.403662 9880 external_mini_cluster.cc:1428] Setting key 93704e082692743301291dc6e1299923
Jun 21 02:44:55 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009895, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM
I20260621 02:44:55.413139 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:55.394807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:58591 (local address 127.9.166.62:37723)
0621 02:44:55.395280 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:55.395287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:55.396350 (+ 1063us) server_negotiation.cc:407] Connection header received
0621 02:44:55.398089 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:55.398094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:55.398165 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:55.398269 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:55.399710 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:55.400535 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:55.401987 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:55.402682 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:55.402836 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:55.406997 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:55.407002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:55.407006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:55.407036 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:55.409583 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:55.410543 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:55.410547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:55.410549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:55.410658 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:55.411157 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:55.411160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:55.411162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:55.411459 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:55.411567 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:55.412449 (+ 882us) server_negotiation.cc:299] Negotiation successful
0621 02:44:55.412700 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":222,"threads_started":1}
I20260621 02:44:55.413780 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:55.395503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.1:58591)
0621 02:44:55.396196 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:55.396232 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:55.397835 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:55.398477 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:55.398484 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:55.398953 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:55.399557 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:55.399575 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:55.400672 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:55.400676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:55.401866 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:55.401877 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:55.402925 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:55.402934 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:55.403136 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:55.403913 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:55.403934 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:55.406801 (+ 2867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:55.409718 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:55.409731 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:55.409751 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:55.410426 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:55.410760 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:55.410769 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:55.410777 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:55.411048 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:55.411584 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:55.411595 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:55.412025 (+ 430us) client_negotiation.cc:769] Sending connection context
0621 02:44:55.412289 (+ 264us) client_negotiation.cc:240] Negotiation successful
0621 02:44:55.412665 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":271,"threads_started":1}
I20260621 02:44:55.415491 16225 heartbeater.cc:347] Connected to a master server at 127.9.166.62:37723
I20260621 02:44:55.415803 16225 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:55.416683 16225 heartbeater.cc:511] Master 127.9.166.62:37723 requested a full tablet report, sending...
I20260621 02:44:55.418816 16036 ts_manager.cc:194] Registered new tserver with Master: df8a57d499a54ac5bc5d7a5579e910cf (127.9.166.1:35761)
I20260621 02:44:55.420552 16036 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:58591
WARNING: no policy specified for oryx/127.9.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.2.keytab.
Entry for principal oryx/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.2.keytab.
I20260621 02:44:55.448102 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:37723
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.2.keytab
--principal=oryx/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:55.759306 16231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:55.759588 16231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:55.759776 16231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:55.766898 16231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:55.767019 16231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:55.767200 16231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:44:55.776605 16231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.2.keytab
--principal=oryx/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:37723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:55.777822 16231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:55.779122 16231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:55.792279 16237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:55.792312 16239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:55.792235 16236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:55.793324 16231 server_base.cc:1034] running on GCE node
I20260621 02:44:55.793906 16231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:55.794790 16231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:55.796006 16231 hybrid_clock.cc:648] HybridClock initialized: now 1782009895795951 us; error 45 us; skew 500 ppm
I20260621 02:44:55.796311 16231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:55 dist-test-slave-hvgn krb5kdc[15991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009895, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:55.799239 16231 init.cc:377] Logged in from keytab as oryx/127.9.166.2@KRBTEST.COM (short username oryx)
I20260621 02:44:55.800617 16231 webserver.cc:460] Webserver started at http://127.9.166.2:46465/ using document root <none> and password file <none>
I20260621 02:44:55.801333 16231 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:55.801479 16231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:55.801769 16231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:55.805028 16231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "57c34184a77044d9ae5d7b754069936b"
format_stamp: "Formatted at 2026-06-21 02:44:55 on dist-test-slave-hvgn"
server_key: "8fc357722cbfa5d371f349240b0ffe74"
server_key_iv: "3ce8f05447a90ad2057d667b0a9db145"
server_key_version: "encryptionkey@0"
I20260621 02:44:55.805958 16231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "57c34184a77044d9ae5d7b754069936b"
format_stamp: "Formatted at 2026-06-21 02:44:55 on dist-test-slave-hvgn"
server_key: "8fc357722cbfa5d371f349240b0ffe74"
server_key_iv: "3ce8f05447a90ad2057d667b0a9db145"
server_key_version: "encryptionkey@0"
I20260621 02:44:55.812117 16231 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260621 02:44:55.816164 16246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:55.817688 16231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:44:55.817840 16231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "57c34184a77044d9ae5d7b754069936b"
format_stamp: "Formatted at 2026-06-21 02:44:55 on dist-test-slave-hvgn"
server_key: "8fc357722cbfa5d371f349240b0ffe74"
server_key_iv: "3ce8f05447a90ad2057d667b0a9db145"
server_key_version: "encryptionkey@0"
I20260621 02:44:55.818049 16231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:55.832240 16231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:55.835891 16231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:55.836126 16231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:55.836944 16231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:55.838268 16231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:55.838335 16231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:55.838425 16231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:55.838479 16231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:55.870958 16231 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:39959
I20260621 02:44:55.871006 16358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:39959 every 8 connection(s)
I20260621 02:44:55.872321 16231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:44:55.875029 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 16231
I20260621 02:44:55.875202 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:44:55.875790 9880 external_mini_cluster.cc:1428] Setting key a5e97d5806958ff95bd9630e2125d45e
Jun 21 02:44:55 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009895, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM
I20260621 02:44:55.893929 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:55.874855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:57761 (local address 127.9.166.62:37723)
0621 02:44:55.875028 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:55.875033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:55.876521 (+ 1488us) server_negotiation.cc:407] Connection header received
0621 02:44:55.878694 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:55.878701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:55.878796 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:55.878907 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:55.880376 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:55.881537 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:55.883057 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:55.883853 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:55.883960 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:55.888091 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:55.888096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:55.888100 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:55.888131 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:55.890646 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:55.891648 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:55.891652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:55.891654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:55.891781 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:55.892261 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:55.892264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:55.892266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:55.892517 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:55.892663 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:55.893332 (+ 669us) server_negotiation.cc:299] Negotiation successful
0621 02:44:55.893567 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 02:44:55.894935 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:55.875550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.2:57761)
0621 02:44:55.876319 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:55.876387 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:55.878427 (+ 2040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:55.879096 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:55.879104 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:55.879672 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:55.880214 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:55.880230 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:55.881692 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:55.881696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:55.882915 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:55.882926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:55.884112 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:55.884121 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:55.884444 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:55.885201 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:55.885236 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:55.887881 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:55.890803 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:55.890814 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:55.890834 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:55.891523 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:55.891896 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:55.891903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:55.891910 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:55.892153 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:55.892638 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:55.892646 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:55.892991 (+ 345us) client_negotiation.cc:769] Sending connection context
0621 02:44:55.893242 (+ 251us) client_negotiation.cc:240] Negotiation successful
0621 02:44:55.893772 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":341,"threads_started":1}
I20260621 02:44:55.896431 16359 heartbeater.cc:347] Connected to a master server at 127.9.166.62:37723
WARNING: no policy specified for oryx/127.9.166.3@KRBTEST.COM; defaulting to no policy
I20260621 02:44:55.896793 16359 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:55.897576 16359 heartbeater.cc:511] Master 127.9.166.62:37723 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.166.3@KRBTEST.COM" created.
I20260621 02:44:55.899067 16036 ts_manager.cc:194] Registered new tserver with Master: 57c34184a77044d9ae5d7b754069936b (127.9.166.2:39959)
I20260621 02:44:55.899943 16036 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:57761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.3.keytab.
Entry for principal oryx/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.3.keytab.
I20260621 02:44:55.920687 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:37723
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.3.keytab
--principal=oryx/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:56.234778 16364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:56.235033 16364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:56.235122 16364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:56.242352 16364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:56.242455 16364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:56.242569 16364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:44:56.251948 16364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.3.keytab
--principal=oryx/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:37723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:56.253084 16364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:56.254264 16364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:56.269158 16372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:56.269203 16369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:56.269124 16370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:56.279259 16364 server_base.cc:1034] running on GCE node
I20260621 02:44:56.279964 16364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:56.280823 16364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:56.282048 16364 hybrid_clock.cc:648] HybridClock initialized: now 1782009896282001 us; error 32 us; skew 500 ppm
I20260621 02:44:56.282382 16364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:56 dist-test-slave-hvgn krb5kdc[15991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009896, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:56.288486 16364 init.cc:377] Logged in from keytab as oryx/127.9.166.3@KRBTEST.COM (short username oryx)
I20260621 02:44:56.289875 16364 webserver.cc:460] Webserver started at http://127.9.166.3:40615/ using document root <none> and password file <none>
I20260621 02:44:56.290539 16364 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:56.290637 16364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:56.290863 16364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:56.293723 16364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "b220186e43e34781af1b15d2d60be62c"
format_stamp: "Formatted at 2026-06-21 02:44:56 on dist-test-slave-hvgn"
server_key: "354e7622d6b7c07057b38f34b3741a37"
server_key_iv: "f6d7f84dbb4779206d995a8ecba2dcbc"
server_key_version: "encryptionkey@0"
I20260621 02:44:56.294438 16364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "b220186e43e34781af1b15d2d60be62c"
format_stamp: "Formatted at 2026-06-21 02:44:56 on dist-test-slave-hvgn"
server_key: "354e7622d6b7c07057b38f34b3741a37"
server_key_iv: "f6d7f84dbb4779206d995a8ecba2dcbc"
server_key_version: "encryptionkey@0"
I20260621 02:44:56.299358 16364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260621 02:44:56.302611 16379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:56.303891 16364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:44:56.304050 16364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "b220186e43e34781af1b15d2d60be62c"
format_stamp: "Formatted at 2026-06-21 02:44:56 on dist-test-slave-hvgn"
server_key: "354e7622d6b7c07057b38f34b3741a37"
server_key_iv: "f6d7f84dbb4779206d995a8ecba2dcbc"
server_key_version: "encryptionkey@0"
I20260621 02:44:56.304234 16364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:56.319633 16364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:56.323871 16364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:56.324157 16364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:56.325026 16364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:56.326277 16364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:56.326345 16364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:56.326500 16364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:56.326555 16364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:56.358971 16364 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:46685
I20260621 02:44:56.359088 16491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:46685 every 8 connection(s)
I20260621 02:44:56.360253 16364 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:44:56.369490 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 16364
I20260621 02:44:56.369658 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:44:56.370103 9880 external_mini_cluster.cc:1428] Setting key 1f645c08fc9dea5a7d99a51e995e301d
Jun 21 02:44:56 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009896, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM
I20260621 02:44:56.379208 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:56.362711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:37995 (local address 127.9.166.62:37723)
0621 02:44:56.362939 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:56.362945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:56.364222 (+ 1277us) server_negotiation.cc:407] Connection header received
0621 02:44:56.365988 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:56.365993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:56.366067 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:56.366154 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:56.367607 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.368328 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.369892 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.370368 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.370509 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:56.374326 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:56.374331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:56.374334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:56.374365 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:56.376834 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:56.377448 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:56.377452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:56.377454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:56.377583 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:56.377921 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:56.377924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:56.377926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:56.378164 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:56.378265 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:56.378836 (+ 571us) server_negotiation.cc:299] Negotiation successful
0621 02:44:56.378981 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:44:56.380121 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:56.363249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.3:37995)
0621 02:44:56.364074 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:56.364111 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:56.365724 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:56.366346 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:56.366355 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:56.366841 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:56.367450 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:56.367469 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.368460 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.368464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:56.369758 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:56.369769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.370583 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.370592 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:56.370823 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:56.371586 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:56.371613 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:56.374142 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:56.376956 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:56.376962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:56.376976 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:56.377352 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:56.377666 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:56.377669 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:56.377671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:56.377827 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:56.378257 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:56.378262 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:56.378548 (+ 286us) client_negotiation.cc:769] Sending connection context
0621 02:44:56.378759 (+ 211us) client_negotiation.cc:240] Negotiation successful
0621 02:44:56.379089 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":365,"threads_started":1}
I20260621 02:44:56.381428 16492 heartbeater.cc:347] Connected to a master server at 127.9.166.62:37723
I20260621 02:44:56.381695 16492 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:56.382356 16492 heartbeater.cc:511] Master 127.9.166.62:37723 requested a full tablet report, sending...
I20260621 02:44:56.383818 16036 ts_manager.cc:194] Registered new tserver with Master: b220186e43e34781af1b15d2d60be62c (127.9.166.3:46685)
I20260621 02:44:56.384871 16036 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:37995
I20260621 02:44:56.385491 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.166.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.166.61@KRBTEST.COM" created.
I20260621 02:44:56.423988 16225 heartbeater.cc:503] Master 127.9.166.62:37723 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.166.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.61.keytab.
Entry for principal oryx/127.9.166.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.61.keytab.
I20260621 02:44:56.429893 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.61:33393
--webserver_interface=127.9.166.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.61.keytab
--principal=oryx/127.9.166.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:37723,127.9.166.61:33393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:44:56.743916 16497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:56.744215 16497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:56.744367 16497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:56.751600 16497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:44:56.751735 16497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:56.751804 16497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:56.751861 16497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:44:56.751904 16497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:44:56.761361 16497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:37723,127.9.166.61:33393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.61.keytab
--principal=oryx/127.9.166.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.61:33393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.166.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
W20260621 02:44:56.761680 16497 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.9.166.62:37723,127.9.166.61:33393'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260621 02:44:56.762639 16497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:56.763968 16497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:56.775358 16502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:56.775336 16503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:56.775482 16505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:56.776253 16497 server_base.cc:1034] running on GCE node
I20260621 02:44:56.776939 16497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:56.778123 16497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:56.779354 16497 hybrid_clock.cc:648] HybridClock initialized: now 1782009896779296 us; error 37 us; skew 500 ppm
I20260621 02:44:56.779691 16497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:56 dist-test-slave-hvgn krb5kdc[15991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009896, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:56.782613 16497 init.cc:377] Logged in from keytab as oryx/127.9.166.61@KRBTEST.COM (short username oryx)
I20260621 02:44:56.783931 16497 webserver.cc:460] Webserver started at http://127.9.166.61:43145/ using document root <none> and password file <none>
I20260621 02:44:56.784523 16497 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:56.784611 16497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:56.784868 16497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:56.787714 16497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/data/instance:
uuid: "bad62c65be0e436b97992c0c818ebf52"
format_stamp: "Formatted at 2026-06-21 02:44:56 on dist-test-slave-hvgn"
server_key: "880d5e599491da4e1358c15560005a47"
server_key_iv: "11a503627ac31fa48ecf014d4523103a"
server_key_version: "encryptionkey@0"
I20260621 02:44:56.788405 16497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/wal/instance:
uuid: "bad62c65be0e436b97992c0c818ebf52"
format_stamp: "Formatted at 2026-06-21 02:44:56 on dist-test-slave-hvgn"
server_key: "880d5e599491da4e1358c15560005a47"
server_key_iv: "11a503627ac31fa48ecf014d4523103a"
server_key_version: "encryptionkey@0"
I20260621 02:44:56.793272 16497 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:44:56.796552 16512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:56.797873 16497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260621 02:44:56.798033 16497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/wal
uuid: "bad62c65be0e436b97992c0c818ebf52"
format_stamp: "Formatted at 2026-06-21 02:44:56 on dist-test-slave-hvgn"
server_key: "880d5e599491da4e1358c15560005a47"
server_key_iv: "11a503627ac31fa48ecf014d4523103a"
server_key_version: "encryptionkey@0"
I20260621 02:44:56.798231 16497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:56.821264 16497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:56.825729 16497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:56.826032 16497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:56.843735 16497 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.61:33393
I20260621 02:44:56.843755 16563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.61:33393 every 8 connection(s)
I20260621 02:44:56.845096 16497 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/data/info.pb
I20260621 02:44:56.846508 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 16497
I20260621 02:44:56.849403 16564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:56.857321 16564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:44:56 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.166.61@KRBTEST.COM
Jun 21 02:44:56 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009896, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.61@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM
I20260621 02:44:56.869297 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:56.848651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.9.166.61:33393)
0621 02:44:56.849370 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:56.849381 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:56.849423 (+ 42us) server_negotiation.cc:407] Connection header received
0621 02:44:56.850988 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:56.851008 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:56.851355 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:56.851743 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:56.853532 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.854851 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.856595 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.857227 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.857444 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:56.861850 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:56.861857 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:56.861867 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:56.861904 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:56.865202 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:56.865845 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:56.865851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:56.865857 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:56.866014 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:56.866400 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:56.866431 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:56.866433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:56.866938 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:56.867269 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:56.867877 (+ 608us) server_negotiation.cc:299] Negotiation successful
0621 02:44:56.868250 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":344,"threads_started":1}
I20260621 02:44:56.875190 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:56.859919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.0.0.1:42360)
0621 02:44:56.860683 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:56.860723 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:56.860919 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:56.861332 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:56.861337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:56.862056 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:56.862390 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:56.862406 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.863903 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.863907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:56.865230 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:56.865243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.866004 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.866011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:56.866179 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:56.866750 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:56.866766 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:56.869034 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:56.872826 (+ 3792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:56.872833 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:56.872837 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:56.873247 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:56.873618 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:56.873620 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:56.873622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:56.873728 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:56.874368 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:56.874373 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:56.874506 (+ 133us) client_negotiation.cc:769] Sending connection context
0621 02:44:56.874656 (+ 150us) client_negotiation.cc:240] Negotiation successful
0621 02:44:56.874921 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":387,"threads_started":1}
I20260621 02:44:56.875569 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:56.859907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42360 (local address 127.9.166.62:37723)
0621 02:44:56.860135 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:56.860141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:56.860806 (+ 665us) server_negotiation.cc:407] Connection header received
0621 02:44:56.861071 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:56.861080 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:56.861174 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:56.861293 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:56.862545 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.863729 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.865371 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.865810 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.865947 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:56.869221 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:56.869225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:56.869229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:56.869261 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:56.872708 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:56.873358 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:56.873363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:56.873367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:56.873516 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:56.873834 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:56.873840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:56.873843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:56.874249 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:56.874400 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:56.874759 (+ 359us) server_negotiation.cc:299] Negotiation successful
0621 02:44:56.874981 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:44:56.877018 16564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:44:56 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009896, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.61@KRBTEST.COM for oryx/127.9.166.61@KRBTEST.COM
I20260621 02:44:56.888296 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:56.877710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:33393 (local address 127.0.0.1:32872)
0621 02:44:56.877928 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:56.877959 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:56.878046 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:56.878282 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:56.878285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:56.878575 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:56.878699 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:56.878706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.879723 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.879727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:56.880563 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:56.880573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.881258 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.881262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:56.881400 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:56.881910 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:56.881926 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:56.883777 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:56.886330 (+ 2553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:56.886333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:56.886335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:56.886682 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:56.886984 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:56.886987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:56.886989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:56.887110 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:56.887645 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:56.887648 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:56.887753 (+ 105us) client_negotiation.cc:769] Sending connection context
0621 02:44:56.887871 (+ 118us) client_negotiation.cc:240] Negotiation successful
0621 02:44:56.888065 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260621 02:44:56.888482 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:56.877710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32872 (local address 127.9.166.61:33393)
0621 02:44:56.877845 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:56.877849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:56.878022 (+ 173us) server_negotiation.cc:407] Connection header received
0621 02:44:56.878129 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:56.878132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:56.878182 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:56.878254 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:56.878824 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.879600 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.880704 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.881141 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.881235 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:56.883944 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:56.883948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:56.883951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:56.883978 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:56.886207 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:56.886773 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:56.886777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:56.886779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:56.886891 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:56.887191 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:56.887194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:56.887196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:56.887550 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:56.887656 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:56.887945 (+ 289us) server_negotiation.cc:299] Negotiation successful
0621 02:44:56.888170 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":47}
I20260621 02:44:56.893013 16564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52: Bootstrap starting.
I20260621 02:44:56.895955 16564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:56.896929 16564 log.cc:824] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:56.899889 16564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52: No bootstrap required, opened a new log
I20260621 02:44:56.902402 16359 heartbeater.cc:503] Master 127.9.166.62:37723 was elected leader, sending a full tablet report...
I20260621 02:44:56.909024 16564 raft_consensus.cc:348] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } }
I20260621 02:44:56.909312 16564 raft_consensus.cc:374] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:56.909380 16564 raft_consensus.cc:729] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bad62c65be0e436b97992c0c818ebf52, State: Initialized, Role: FOLLOWER
I20260621 02:44:56.909853 16564 consensus_queue.cc:260] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } }
I20260621 02:44:56.910631 16570 sys_catalog.cc:455] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } } }
I20260621 02:44:56.910964 16570 sys_catalog.cc:458] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:44:56.911597 16564 sys_catalog.cc:564] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: configured and running, proceeding with master startup.
W20260621 02:44:56.920022 16581 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:44:56.920154 16581 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:44:56.921836 16497 master_runner.cc:428] Detected that this master bad62c65be0e436b97992c0c818ebf52 is joining an existing cluster
I20260621 02:44:56.921916 16497 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.166.61:33393
I20260621 02:44:56.952498 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:56.929943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42376 (local address 127.9.166.62:37723)
0621 02:44:56.930161 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:56.930169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:56.936000 (+ 5831us) server_negotiation.cc:407] Connection header received
0621 02:44:56.936268 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:56.936274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:56.936361 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:56.936431 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:56.937916 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.939040 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.941133 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.941773 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.941869 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:56.945387 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:56.945395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:56.945399 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:56.945439 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:56.949228 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:56.950036 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:56.950042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:56.950046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:56.950200 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:56.950669 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:56.950675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:56.950678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:56.951020 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:56.951153 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:56.951883 (+ 730us) server_negotiation.cc:299] Negotiation successful
0621 02:44:56.952136 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 02:44:56.952847 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:56.929412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.9.166.61:33393)
0621 02:44:56.929624 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:56.929633 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:56.936651 (+ 7018us) server_negotiation.cc:407] Connection header received
0621 02:44:56.936842 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:56.936848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:56.936927 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:56.937022 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:56.938525 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.939788 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.941949 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.942715 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.942836 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:56.945665 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:56.945673 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:56.945681 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:56.945718 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:56.949520 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:56.950419 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:56.950427 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:56.950434 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:56.950619 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:56.951081 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:56.951089 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:56.951094 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:56.951574 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:56.951720 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:56.952223 (+ 503us) server_negotiation.cc:299] Negotiation successful
0621 02:44:56.952450 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":70}
WARNING: no policy specified for oryx/127.9.166.60@KRBTEST.COM; defaulting to no policy
I20260621 02:44:56.953889 16587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:56.929434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.0.0.1:42376)
0621 02:44:56.935893 (+ 6459us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:56.935913 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:56.936110 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:56.936653 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:56.936657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:56.937315 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:56.937750 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:56.937764 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.939230 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.939238 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:56.940973 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:56.940987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.942147 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.942155 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:56.943124 (+ 969us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:56.943819 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:56.943846 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:56.945189 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:56.949386 (+ 4197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:56.949391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:56.949395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:56.949898 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:56.950345 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:56.950349 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:56.950353 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:56.950519 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:56.951233 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:56.951238 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:56.951534 (+ 296us) client_negotiation.cc:769] Sending connection context
0621 02:44:56.953042 (+ 1508us) client_negotiation.cc:240] Negotiation successful
0621 02:44:56.953439 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6314,"mutex_wait_us":49,"thread_start_us":6179,"threads_started":1}
I20260621 02:44:56.954206 16588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:56.929592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:33393 (local address 127.0.0.1:32880)
0621 02:44:56.936554 (+ 6962us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:56.936569 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:56.936720 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:56.937186 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:56.937192 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:56.938018 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:56.938374 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:56.938388 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.940054 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.940061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:56.941801 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:56.941817 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.942923 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.942931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:56.943367 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:56.944027 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:56.944055 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:56.945488 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:56.949662 (+ 4174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:56.949669 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:56.949676 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:56.950286 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:56.950765 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:56.950772 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:56.950778 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:56.950959 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:56.951803 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:56.951811 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:56.951967 (+ 156us) client_negotiation.cc:769] Sending connection context
0621 02:44:56.953289 (+ 1322us) client_negotiation.cc:240] Negotiation successful
0621 02:44:56.953623 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6833,"mutex_wait_us":1,"thread_start_us":6557,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.166.60@KRBTEST.COM" created.
Jun 21 02:44:56 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009894, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.62@KRBTEST.COM for oryx/127.9.166.61@KRBTEST.COM
I20260621 02:44:56.976414 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:56.960932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32884 (local address 127.9.166.61:33393)
0621 02:44:56.961757 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:56.961765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:56.961784 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:44:56.961863 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:56.961870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:56.961980 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:56.962075 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:56.963810 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.965078 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.966469 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.966919 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.967072 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:56.970379 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:56.970388 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:56.970396 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:56.970452 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:56.973448 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:56.974309 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:56.974319 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:56.974324 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:56.974481 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:56.974837 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:56.974843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:56.974846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:56.975155 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:56.975308 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:56.975813 (+ 505us) server_negotiation.cc:299] Negotiation successful
0621 02:44:56.976038 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694}
I20260621 02:44:56.976480 16590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:56.960491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:33393 (local address 127.0.0.1:32884)
0621 02:44:56.961091 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:56.961138 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:56.961294 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:56.962300 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:56.962309 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:56.963246 (+ 937us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:56.963591 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:56.963607 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.965266 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.965274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:56.966299 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:56.966310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:56.967090 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:56.967096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:56.967316 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:56.968090 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:56.968119 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:56.970195 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:56.973618 (+ 3423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:56.973631 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:56.973639 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:56.974158 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:56.974594 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:56.974600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:56.974603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:56.974731 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:56.975289 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:56.975297 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:56.975549 (+ 252us) client_negotiation.cc:769] Sending connection context
0621 02:44:56.975736 (+ 187us) client_negotiation.cc:240] Negotiation successful
0621 02:44:56.976039 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":484,"threads_started":1}
I20260621 02:44:56.978597 16035 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.166.61:33393: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bad62c65be0e436b97992c0c818ebf52"
member_type: NON_VOTER
last_known_addr {
host: "127.9.166.61"
port: 33393
}
attrs {
promote: true
}
}
dest_uuid: "2296b5aeb8f34f529d7886dae3a889cf"
cas_config_opid_index: -1
I20260621 02:44:56.979610 16035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } }
I20260621 02:44:56.983773 16594 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER bad62c65be0e436b97992c0c818ebf52 (127.9.166.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } } }
I20260621 02:44:56.984915 16539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.166.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.60.keytab.
Entry for principal oryx/127.9.166.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.60.keytab.
I20260621 02:44:56.990585 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.60:35903
--webserver_interface=127.9.166.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.60.keytab
--principal=oryx/127.9.166.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:37723,127.9.166.61:33393,127.9.166.60:35903
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:44:56.990994 16539 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 1 FOLLOWER]: Refusing update from remote peer 2296b5aeb8f34f529d7886dae3a889cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260621 02:44:56.993448 16593 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:56.995749 16594 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.166.61:33393
I20260621 02:44:56.996730 16497 master.cc:537] Master@127.9.166.61:33393 shutting down...
I20260621 02:44:56.997264 16594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } } }
I20260621 02:44:56.997608 16594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:57.024500 16539 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER bad62c65be0e436b97992c0c818ebf52 (127.9.166.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } } }
I20260621 02:44:57.047158 16603 sys_catalog.cc:455] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } } }
I20260621 02:44:57.047747 16599 sys_catalog.cc:455] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } } }
I20260621 02:44:57.048147 16599 sys_catalog.cc:458] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: This master's current role is: LEARNER
I20260621 02:44:57.048539 16570 sys_catalog.cc:455] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2296b5aeb8f34f529d7886dae3a889cf. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } } }
I20260621 02:44:57.048761 16570 sys_catalog.cc:458] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: This master's current role is: LEARNER
I20260621 02:44:57.047747 16603 sys_catalog.cc:458] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: This master's current role is: LEARNER
I20260621 02:44:57.050823 16497 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 1 LEARNER]: Raft consensus shutting down.
I20260621 02:44:57.051237 16497 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 1 LEARNER]: Raft consensus is shut down!
I20260621 02:44:57.051416 16497 tablet_replica.cc:330] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52: stopping tablet replica
I20260621 02:44:57.058424 16497 master.cc:552] Master@127.9.166.61:33393 shutdown complete.
I20260621 02:44:57.058514 16497 master_runner.cc:305] Clearing existing system tablet
I20260621 02:44:57.058972 16497 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:57.059074 16497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:57.061985 16497 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260621 02:44:57.063616 16605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:57.063983 16497 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:57.064265 16497 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 02:44:57.064353 16497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/wal
uuid: "bad62c65be0e436b97992c0c818ebf52"
format_stamp: "Formatted at 2026-06-21 02:44:56 on dist-test-slave-hvgn"
server_key: "880d5e599491da4e1358c15560005a47"
server_key_iv: "11a503627ac31fa48ecf014d4523103a"
server_key_version: "encryptionkey@0"
I20260621 02:44:57.068732 16497 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:57.075369 16497 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260621 02:44:57.075578 16497 log.cc:1196] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260621 02:44:57.076040 16497 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52: Deleting consensus metadata
I20260621 02:44:57.076292 16497 master_runner.cc:315] Copying system tablet from 127.9.166.62:37723
I20260621 02:44:57.078735 16497 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52: tablet copy: Beginning tablet copy session from remote peer at address 127.9.166.62:37723
I20260621 02:44:57.098296 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.079632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42386 (local address 127.9.166.62:37723)
0621 02:44:57.079827 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:57.079833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:57.083931 (+ 4098us) server_negotiation.cc:407] Connection header received
0621 02:44:57.084155 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:57.084164 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:57.084274 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:57.084388 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:57.085415 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.086668 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.088554 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.089173 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.089302 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.091658 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:57.091665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:57.091670 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:57.091710 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:57.095245 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.096025 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.096031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.096034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.096186 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.096617 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.096624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.096630 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.097087 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:57.097259 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:57.097726 (+ 467us) server_negotiation.cc:299] Negotiation successful
0621 02:44:57.097961 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:44:57.098395 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.079441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.0.0.1:42386)
0621 02:44:57.083792 (+ 4351us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:57.083816 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:57.083984 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:57.084427 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:57.084433 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:57.084935 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:57.085217 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:57.085229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.086840 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.086846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:57.088398 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:57.088412 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.089326 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.089332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:57.089631 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.090338 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:57.090361 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:57.091456 (+ 1095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:57.095380 (+ 3924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:57.095406 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:57.095411 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:57.095890 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:57.096296 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:57.096303 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:57.096312 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:57.096482 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:57.097220 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:57.097225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:57.097418 (+ 193us) client_negotiation.cc:769] Sending connection context
0621 02:44:57.097579 (+ 161us) client_negotiation.cc:240] Negotiation successful
0621 02:44:57.097921 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4219,"thread_start_us":4108,"threads_started":1}
I20260621 02:44:57.099514 16056 tablet_copy_service.cc:140] P 2296b5aeb8f34f529d7886dae3a889cf: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bad62c65be0e436b97992c0c818ebf52 ({username='oryx', principal='oryx/127.9.166.61@KRBTEST.COM'} at 127.0.0.1:42386)
I20260621 02:44:57.099771 16056 tablet_copy_service.cc:161] P 2296b5aeb8f34f529d7886dae3a889cf: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bad62c65be0e436b97992c0c818ebf52 at {username='oryx', principal='oryx/127.9.166.61@KRBTEST.COM'} at 127.0.0.1:42386: session id = bad62c65be0e436b97992c0c818ebf52-00000000000000000000000000000000
I20260621 02:44:57.104079 16056 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 02:44:57.106253 16497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260621 02:44:57.112805 16497 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52: tablet copy: Starting download of 0 data blocks...
I20260621 02:44:57.113173 16497 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52: tablet copy: Starting download of 1 WAL segments...
I20260621 02:44:57.115788 16497 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 02:44:57.119917 16056 tablet_copy_service.cc:342] P 2296b5aeb8f34f529d7886dae3a889cf: Request end of tablet copy session bad62c65be0e436b97992c0c818ebf52-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.166.61@KRBTEST.COM'} at 127.0.0.1:42386
I20260621 02:44:57.120123 16056 tablet_copy_service.cc:433] P 2296b5aeb8f34f529d7886dae3a889cf: ending tablet copy session bad62c65be0e436b97992c0c818ebf52-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bad62c65be0e436b97992c0c818ebf52
I20260621 02:44:57.132416 16497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:57.134896 16612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:57.136485 16613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:57.147616 16497 server_base.cc:1034] running on GCE node
W20260621 02:44:57.147706 16615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:57.148231 16497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:57.148890 16497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:57.150157 16497 hybrid_clock.cc:648] HybridClock initialized: now 1782009897150088 us; error 33 us; skew 500 ppm
I20260621 02:44:57.150453 16497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:57 dist-test-slave-hvgn krb5kdc[15991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009897, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:57.152603 16497 init.cc:377] Logged in from keytab as oryx/127.9.166.61@KRBTEST.COM (short username oryx)
I20260621 02:44:57.153831 16497 webserver.cc:460] Webserver started at http://127.9.166.61:38633/ using document root <none> and password file <none>
I20260621 02:44:57.154191 16497 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:57.154269 16497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:57.157199 16497 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260621 02:44:57.159055 16622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:57.159479 16497 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260621 02:44:57.159595 16497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/wal
uuid: "bad62c65be0e436b97992c0c818ebf52"
format_stamp: "Formatted at 2026-06-21 02:44:56 on dist-test-slave-hvgn"
server_key: "880d5e599491da4e1358c15560005a47"
server_key_iv: "11a503627ac31fa48ecf014d4523103a"
server_key_version: "encryptionkey@0"
I20260621 02:44:57.159757 16497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:57.177357 16497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:57.181528 16497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:57.214234 16497 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.61:33393
I20260621 02:44:57.214375 16673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.61:33393 every 8 connection(s)
I20260621 02:44:57.215197 16497 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-1/data/info.pb
I20260621 02:44:57.220772 16674 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:44:57.225515 16674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52: Bootstrap starting.
I20260621 02:44:57.238873 16674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:44:57.239439 16674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52: Bootstrap complete.
I20260621 02:44:57.240386 16674 raft_consensus.cc:348] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } }
I20260621 02:44:57.240623 16674 raft_consensus.cc:729] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bad62c65be0e436b97992c0c818ebf52, State: Initialized, Role: LEARNER
I20260621 02:44:57.240932 16674 consensus_queue.cc:260] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } }
I20260621 02:44:57.241834 16677 sys_catalog.cc:455] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } } }
I20260621 02:44:57.242223 16677 sys_catalog.cc:458] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: This master's current role is: LEARNER
I20260621 02:44:57.242261 16674 sys_catalog.cc:564] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:44:57.252130 16688 catalog_manager.cc:1259] Loaded cluster ID: c80c61a5840d48a68a453a9bbfc9acb9
I20260621 02:44:57.252223 16688 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52: loading cluster ID for follower catalog manager: success
I20260621 02:44:57.255462 16688 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52: acquiring CA information for follower catalog manager: success
I20260621 02:44:57.257849 16688 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 02:44:57.370770 16595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:57.371184 16595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:57.371431 16595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:57.378566 16595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:44:57.378693 16595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:57.378789 16595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:57.378881 16595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:44:57.378942 16595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:44:57.387367 16492 heartbeater.cc:503] Master 127.9.166.62:37723 was elected leader, sending a full tablet report...
I20260621 02:44:57.389122 16595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:37723,127.9.166.61:33393,127.9.166.60:35903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.60.keytab
--principal=oryx/127.9.166.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.60:35903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.166.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:57.390618 16595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:57.392201 16595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:57.404495 16696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:57.404546 16694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:57.404543 16693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:57.406323 16595 server_base.cc:1034] running on GCE node
I20260621 02:44:57.407128 16595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:57.408589 16595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:57.409897 16595 hybrid_clock.cc:648] HybridClock initialized: now 1782009897409840 us; error 46 us; skew 500 ppm
I20260621 02:44:57.410261 16595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:57 dist-test-slave-hvgn krb5kdc[15991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009897, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:57.413496 16595 init.cc:377] Logged in from keytab as oryx/127.9.166.60@KRBTEST.COM (short username oryx)
I20260621 02:44:57.415000 16595 webserver.cc:460] Webserver started at http://127.9.166.60:34577/ using document root <none> and password file <none>
I20260621 02:44:57.415673 16595 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:57.415768 16595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:57.416003 16595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:44:57.418717 16595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/data/instance:
uuid: "9ea8bd55103d4ab7ba775a30111cd5a0"
format_stamp: "Formatted at 2026-06-21 02:44:57 on dist-test-slave-hvgn"
server_key: "eb85fb1b9b9d1c53a5c657f4bca82573"
server_key_iv: "e92f4dd56ab427976d5f8f6a64fa964b"
server_key_version: "encryptionkey@0"
I20260621 02:44:57.419450 16595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/wal/instance:
uuid: "9ea8bd55103d4ab7ba775a30111cd5a0"
format_stamp: "Formatted at 2026-06-21 02:44:57 on dist-test-slave-hvgn"
server_key: "eb85fb1b9b9d1c53a5c657f4bca82573"
server_key_iv: "e92f4dd56ab427976d5f8f6a64fa964b"
server_key_version: "encryptionkey@0"
I20260621 02:44:57.424311 16595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 02:44:57.427564 16703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:57.428800 16595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:44:57.428961 16595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/wal
uuid: "9ea8bd55103d4ab7ba775a30111cd5a0"
format_stamp: "Formatted at 2026-06-21 02:44:57 on dist-test-slave-hvgn"
server_key: "eb85fb1b9b9d1c53a5c657f4bca82573"
server_key_iv: "e92f4dd56ab427976d5f8f6a64fa964b"
server_key_version: "encryptionkey@0"
I20260621 02:44:57.429152 16595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:57.442545 16595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:57.446601 16595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:57.446849 16595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:57.464267 16595 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.60:35903
I20260621 02:44:57.464314 16754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.60:35903 every 8 connection(s)
I20260621 02:44:57.465679 16595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/data/info.pb
I20260621 02:44:57.469098 16755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:57.475466 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 16595
I20260621 02:44:57.475180 16755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:44:57 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009897, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.60@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM
Jun 21 02:44:57 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.166.60@KRBTEST.COM
I20260621 02:44:57.494992 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.478101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42400 (local address 127.9.166.62:37723)
0621 02:44:57.478287 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:57.478293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:57.479163 (+ 870us) server_negotiation.cc:407] Connection header received
0621 02:44:57.480410 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:57.480418 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:57.480499 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:57.480615 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:57.482291 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.482995 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.484534 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.484973 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.485116 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.488571 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:57.488576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:57.488579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:57.488612 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:57.491990 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.492747 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.492753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.492758 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.492910 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.493325 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.493334 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.493336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.493614 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:57.493765 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:57.494432 (+ 667us) server_negotiation.cc:299] Negotiation successful
0621 02:44:57.494662 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 02:44:57.496119 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.478094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.0.0.1:42400)
0621 02:44:57.478994 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:57.479046 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:57.480162 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:57.481104 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:57.481109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:57.481666 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:57.482100 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:57.482126 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.483159 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.483164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:57.484397 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:57.484409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.485207 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.485213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:57.485350 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.486027 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:57.486063 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:57.488379 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:57.492122 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:57.492130 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:57.492138 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:57.492615 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:57.493029 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:57.493033 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:57.493035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:57.493196 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:57.493737 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:57.493744 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:57.494062 (+ 318us) client_negotiation.cc:769] Sending connection context
0621 02:44:57.494324 (+ 262us) client_negotiation.cc:240] Negotiation successful
0621 02:44:57.494868 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":1,"spinlock_wait_cycles":12672,"thread_start_us":502,"threads_started":1}
I20260621 02:44:57.496137 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.477615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43490 (local address 127.9.166.60:35903)
0621 02:44:57.478385 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:57.478395 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:57.478430 (+ 35us) server_negotiation.cc:407] Connection header received
0621 02:44:57.480152 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:57.480171 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:57.480448 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:57.480668 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:57.481993 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.483275 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.484581 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.485089 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.485310 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.489718 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:57.489728 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:57.489745 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:57.489785 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:57.493103 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.493728 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.493737 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.493740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.493872 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.494289 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.494294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.494297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.494609 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:57.494959 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:57.495230 (+ 271us) server_negotiation.cc:299] Negotiation successful
0621 02:44:57.495500 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":586,"thread_start_us":392,"threads_started":1}
I20260621 02:44:57.497583 16755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:44:57 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009897, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.60@KRBTEST.COM for oryx/127.9.166.61@KRBTEST.COM
I20260621 02:44:57.510283 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.498352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:33393 (local address 127.0.0.1:32898)
0621 02:44:57.498502 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:57.498521 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:57.498631 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:57.499682 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:57.499685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:57.499988 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:57.500114 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:57.500121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.501352 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.501356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:57.502328 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:57.502338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.502960 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.502964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:57.503083 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.503660 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:57.503677 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:57.505478 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:57.508471 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:57.508474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:57.508477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:57.508827 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:57.509153 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:57.509155 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:57.509158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:57.509268 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:57.509679 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:57.509682 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:57.509786 (+ 104us) client_negotiation.cc:769] Sending connection context
0621 02:44:57.509885 (+ 99us) client_negotiation.cc:240] Negotiation successful
0621 02:44:57.510063 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260621 02:44:57.510576 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.498434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32898 (local address 127.9.166.61:33393)
0621 02:44:57.499322 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:57.499326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:57.499341 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:44:57.499443 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:57.499447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:57.499503 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:57.499593 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:57.500256 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.501214 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.502457 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.502847 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.502940 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.505686 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:57.505691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:57.505694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:57.505724 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:57.508359 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.508917 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.508920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.508922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.509062 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.509348 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.509351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.509353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.509592 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:57.509703 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:57.510059 (+ 356us) server_negotiation.cc:299] Negotiation successful
0621 02:44:57.510212 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":622,"threads_started":1}
I20260621 02:44:57.511564 16755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:44:57 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009897, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.60@KRBTEST.COM for oryx/127.9.166.60@KRBTEST.COM
I20260621 02:44:57.523237 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.512307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:35903 (local address 127.0.0.1:43502)
0621 02:44:57.512490 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:57.512509 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:57.512654 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:57.512913 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:57.512917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:57.513268 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:57.513387 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:57.513402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.514324 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.514332 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:57.515226 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:57.515237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.515902 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.515907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:57.516116 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.516643 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:57.516658 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:57.518409 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:57.520903 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:57.520907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:57.520911 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:57.521227 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:57.521567 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:57.521571 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:57.521575 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:57.521706 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:57.522315 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:57.522320 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:57.522483 (+ 163us) client_negotiation.cc:769] Sending connection context
0621 02:44:57.522683 (+ 200us) client_negotiation.cc:240] Negotiation successful
0621 02:44:57.522951 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 02:44:57.523276 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.512343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43502 (local address 127.9.166.60:35903)
0621 02:44:57.512490 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:57.512497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:57.512598 (+ 101us) server_negotiation.cc:407] Connection header received
0621 02:44:57.512753 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:57.512758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:57.512812 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:57.512900 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:57.513515 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.514213 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.515348 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.515773 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.515897 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.518570 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:57.518575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:57.518578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:57.518602 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:57.520777 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.521329 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.521337 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.521340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.521454 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.521829 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.521833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.521835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.522083 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:57.522203 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:57.522760 (+ 557us) server_negotiation.cc:299] Negotiation successful
0621 02:44:57.523055 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":56}
I20260621 02:44:57.527792 16755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: Bootstrap starting.
I20260621 02:44:57.531000 16755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:57.532135 16755 log.cc:824] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:57.534804 16755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: No bootstrap required, opened a new log
I20260621 02:44:57.541990 16755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } }
I20260621 02:44:57.542322 16755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:57.542446 16755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea8bd55103d4ab7ba775a30111cd5a0, State: Initialized, Role: FOLLOWER
I20260621 02:44:57.543138 16755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } }
I20260621 02:44:57.544085 16762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } } }
I20260621 02:44:57.544472 16762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:44:57.545125 16755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:44:57.546192 16762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:44:57.546460 16762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } }
I20260621 02:44:57.548208 16762 leader_election.cc:288] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2296b5aeb8f34f529d7886dae3a889cf (127.9.166.62:37723), bad62c65be0e436b97992c0c818ebf52 (127.9.166.61:33393)
I20260621 02:44:57.548976 16046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2296b5aeb8f34f529d7886dae3a889cf" is_pre_election: true
I20260621 02:44:57.548930 16649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bad62c65be0e436b97992c0c818ebf52" is_pre_election: true
I20260621 02:44:57.549263 16649 raft_consensus.cc:1799] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 1 LEARNER]: Handling vote request from an unknown peer 9ea8bd55103d4ab7ba775a30111cd5a0
I20260621 02:44:57.549535 16649 raft_consensus.cc:2396] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 9ea8bd55103d4ab7ba775a30111cd5a0 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260621 02:44:57.550225 16707 leader_election.cc:302] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea8bd55103d4ab7ba775a30111cd5a0; no voters: 2296b5aeb8f34f529d7886dae3a889cf, bad62c65be0e436b97992c0c818ebf52
I20260621 02:44:57.550740 16762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:57.553542 16762 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260621 02:44:57.558468 16773 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:44:57.559147 16773 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:44:57.561992 16595 master_runner.cc:428] Detected that this master 9ea8bd55103d4ab7ba775a30111cd5a0 is joining an existing cluster
I20260621 02:44:57.562114 16595 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.166.60:35903
I20260621 02:44:57.595093 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.566261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32912 (local address 127.9.166.61:33393)
0621 02:44:57.566437 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:57.566443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:57.579095 (+ 12652us) server_negotiation.cc:407] Connection header received
0621 02:44:57.579231 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:57.579237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:57.579331 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:57.579472 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:57.581045 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.582145 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.584421 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.585079 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.585181 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.587973 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:57.587979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:57.587984 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:57.588031 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:57.591502 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.592474 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.592479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.592483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.592644 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.593262 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.593268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.593271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.593618 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:57.593756 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:57.594390 (+ 634us) server_negotiation.cc:299] Negotiation successful
0621 02:44:57.594643 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:44:57.595757 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.567089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43508 (local address 127.9.166.60:35903)
0621 02:44:57.567219 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:57.567225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:57.577977 (+ 10752us) server_negotiation.cc:407] Connection header received
0621 02:44:57.578169 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:57.578174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:57.578231 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:57.578286 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:57.580443 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.581525 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.583599 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.584420 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.584525 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.587770 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:57.587776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:57.587781 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:57.587819 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:57.592537 (+ 4718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.593461 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.593465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.593468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.593601 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.594025 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.594034 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.594036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.594384 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:57.594504 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:57.594984 (+ 480us) server_negotiation.cc:299] Negotiation successful
0621 02:44:57.595263 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":30,"spinlock_wait_cycles":13056}
I20260621 02:44:57.596513 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.566740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.9.166.62:37723)
0621 02:44:57.567267 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:57.567275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:57.577613 (+ 10338us) server_negotiation.cc:407] Connection header received
0621 02:44:57.578092 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:57.578101 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:57.578188 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:57.578261 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:57.580020 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.581204 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.583472 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.584134 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.584241 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.587750 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:57.587757 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:57.587762 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:57.587804 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:57.591710 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.592661 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.592666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.592669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.592830 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.593275 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.593282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.593289 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.593675 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:57.593800 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:57.595324 (+ 1524us) server_negotiation.cc:299] Negotiation successful
0621 02:44:57.596004 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407}
I20260621 02:44:57.596953 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.566312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:35903 (local address 127.0.0.1:43508)
0621 02:44:57.577877 (+ 11565us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:57.577893 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:57.578057 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:57.578452 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:57.578457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:57.579970 (+ 1513us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:57.580308 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:57.580319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.581748 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.581759 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:57.583480 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:57.583491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.584784 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.584789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:57.585011 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.585839 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:57.585881 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:57.587533 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:57.592651 (+ 5118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:57.592655 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:57.592658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:57.593359 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:57.593738 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:57.593741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:57.593745 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:57.593899 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:57.594572 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:57.594580 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:57.594745 (+ 165us) client_negotiation.cc:769] Sending connection context
0621 02:44:57.595949 (+ 1204us) client_negotiation.cc:240] Negotiation successful
0621 02:44:57.596247 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11445,"mutex_wait_us":18,"spinlock_wait_cycles":13440,"thread_start_us":10830,"threads_started":1}
I20260621 02:44:57.600335 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.566236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.0.0.1:42414)
0621 02:44:57.577485 (+ 11249us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:57.577509 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:57.577911 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:57.578452 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:57.578457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:57.579299 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:57.579836 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:57.579852 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.581374 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.581379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:57.583262 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:57.583276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.584548 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.584553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:57.584721 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.585839 (+ 1118us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:57.585863 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:57.587414 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:57.591870 (+ 4456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:57.591878 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:57.591884 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:57.592528 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:57.592948 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:57.592956 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:57.592960 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:57.593144 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:57.593867 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:57.593876 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:57.594117 (+ 241us) client_negotiation.cc:769] Sending connection context
0621 02:44:57.596782 (+ 2665us) client_negotiation.cc:240] Negotiation successful
0621 02:44:57.599747 (+ 2965us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11055,"mutex_wait_us":24,"spinlock_wait_cycles":1536,"thread_start_us":11386,"threads_started":1}
I20260621 02:44:57.601104 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.566761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:33393 (local address 127.0.0.1:32912)
0621 02:44:57.578969 (+ 12208us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:57.578991 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:57.579117 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:57.579662 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:57.579667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:57.580465 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:57.580862 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:57.580875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.582331 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.582337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:57.584259 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:57.584273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.585455 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.585461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:57.585634 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.586248 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:57.586272 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:57.587749 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:57.591669 (+ 3920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:57.591675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:57.591679 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:57.592338 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:57.592791 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:57.592796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:57.592800 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:57.593123 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:57.593832 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:57.593836 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:57.594057 (+ 221us) client_negotiation.cc:769] Sending connection context
0621 02:44:57.598575 (+ 4518us) client_negotiation.cc:240] Negotiation successful
0621 02:44:57.599952 (+ 1377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12045,"mutex_wait_us":1032,"thread_start_us":11886,"threads_started":1}
I20260621 02:44:57.604031 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 16097
I20260621 02:44:57.620126 16784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } } }
I20260621 02:44:57.620530 16784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: This master's current role is: LEADER
Jun 21 02:44:57 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009894, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.62@KRBTEST.COM for oryx/127.9.166.60@KRBTEST.COM
Jun 21 02:44:57 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.166.1@KRBTEST.COM
I20260621 02:44:57.632611 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.609042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.9.166.60:35903)
0621 02:44:57.609281 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:57.609300 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:57.609365 (+ 65us) server_negotiation.cc:407] Connection header received
0621 02:44:57.609586 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:57.609598 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:57.609692 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:57.609767 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:57.610971 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.612102 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.614144 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.615107 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.615252 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.622637 (+ 7385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:57.622647 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:57.622654 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:57.622703 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:57.627404 (+ 4701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.630063 (+ 2659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.630070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.630075 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.630223 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.630696 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.630701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.630704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.631040 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:57.631173 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:57.631993 (+ 820us) server_negotiation.cc:299] Negotiation successful
0621 02:44:57.632249 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 02:44:57.633469 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.608614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:35903 (local address 127.0.0.1:43518)
0621 02:44:57.609215 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:57.609235 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:57.609354 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:57.610028 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:57.610032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:57.610568 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:57.610792 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:57.610805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.612324 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.612339 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:57.613962 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:57.613982 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.616587 (+ 2605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.616596 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:57.616863 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.618826 (+ 1963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:57.618859 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:57.622383 (+ 3524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:57.629340 (+ 6957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:57.629345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:57.629350 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:57.629924 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:57.630352 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:57.630357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:57.630361 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:57.630556 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:57.631250 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:57.631256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:57.631454 (+ 198us) client_negotiation.cc:769] Sending connection context
0621 02:44:57.631611 (+ 157us) client_negotiation.cc:240] Negotiation successful
0621 02:44:57.632992 (+ 1381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":1,"thread_start_us":815,"threads_started":1}
I20260621 02:44:57.635766 16035 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.166.60:35903: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0"
member_type: NON_VOTER
last_known_addr {
host: "127.9.166.60"
port: 35903
}
attrs {
promote: true
}
}
dest_uuid: "2296b5aeb8f34f529d7886dae3a889cf"
cas_config_opid_index: 5
I20260621 02:44:57.636487 16035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: NON_VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: true } }
I20260621 02:44:57.636677 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.614091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.9.166.61:33393)
0621 02:44:57.614273 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:57.614278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:57.627171 (+ 12893us) server_negotiation.cc:407] Connection header received
0621 02:44:57.627614 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:57.627621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:57.627721 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:57.627806 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:57.628670 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.629781 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.633174 (+ 3393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.634284 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.634364 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.634519 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:57.636214 (+ 1695us) server_negotiation.cc:299] Negotiation successful
0621 02:44:57.636383 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:44:57.637429 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.613615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:33393 (local address 127.0.0.1:32920)
0621 02:44:57.627065 (+ 13450us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:57.627085 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:57.627451 (+ 366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:57.628047 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:57.628052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:57.628078 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:57.628460 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:57.628473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.629964 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.629972 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:57.632992 (+ 3020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:57.633005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.635718 (+ 2713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.635727 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:57.635958 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.636048 (+ 90us) client_negotiation.cc:769] Sending connection context
0621 02:44:57.636924 (+ 876us) client_negotiation.cc:240] Negotiation successful
0621 02:44:57.637074 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13288,"mutex_wait_us":763,"thread_start_us":13576,"threads_started":1}
I20260621 02:44:57.638521 16785 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9ea8bd55103d4ab7ba775a30111cd5a0 (127.9.166.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: NON_VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: true } } }
I20260621 02:44:57.639964 16730 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 1 FOLLOWER]: Refusing update from remote peer 2296b5aeb8f34f529d7886dae3a889cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260621 02:44:57.640906 16677 sys_catalog.cc:455] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2296b5aeb8f34f529d7886dae3a889cf. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } } }
I20260621 02:44:57.641170 16677 sys_catalog.cc:458] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: This master's current role is: LEARNER
I20260621 02:44:57.641386 16648 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 1 LEARNER]: Refusing update from remote peer 2296b5aeb8f34f529d7886dae3a889cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260621 02:44:57.642875 16784 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: NON_VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:44:57.647886 16789 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260621 02:44:57.648275 16785 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.166.60:35903
I20260621 02:44:57.648353 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.607757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45576 (local address 127.9.166.1:35761)
0621 02:44:57.608395 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:57.608408 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:57.608447 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:44:57.608522 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:57.608531 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:57.608829 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:57.609010 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:57.616642 (+ 7632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.618613 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.620992 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.622065 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.626657 (+ 4592us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.629411 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:57.629429 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:57.629455 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:57.629524 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:57.634774 (+ 5250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.644427 (+ 9653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.644442 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.644468 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.644704 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.645338 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.645346 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.645352 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.646080 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:57.646391 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:57.647488 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0621 02:44:57.647826 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":943,"threads_started":1}
I20260621 02:44:57.649230 16595 master.cc:537] Master@127.9.166.60:35903 shutting down...
I20260621 02:44:57.650765 16220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:57.661772 16784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: NON_VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: true } } }
I20260621 02:44:57.662093 16784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:57.666260 16730 raft_consensus.cc:858] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: NON_VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: true } }
I20260621 02:44:57.671708 16648 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9ea8bd55103d4ab7ba775a30111cd5a0 (127.9.166.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: NON_VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: true } } }
I20260621 02:44:57.675953 16677 sys_catalog.cc:455] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: NON_VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: true } } }
I20260621 02:44:57.676230 16677 sys_catalog.cc:458] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: This master's current role is: LEARNER
I20260621 02:44:57.676829 16784 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf: attempting to promote NON_VOTER bad62c65be0e436b97992c0c818ebf52 to VOTER
I20260621 02:44:57.677541 16784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: NON_VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: true } }
W20260621 02:44:57.666464 16730 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: NON_VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: true } }
I20260621 02:44:57.681107 16649 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 1 LEARNER]: Refusing update from remote peer 2296b5aeb8f34f529d7886dae3a889cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260621 02:44:57.682281 16022 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260621 02:44:57.682824 16786 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260621 02:44:57.684130 16022 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf -> Peer 9ea8bd55103d4ab7ba775a30111cd5a0 (127.9.166.60:35903): Couldn't send request to peer 9ea8bd55103d4ab7ba775a30111cd5a0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260621 02:44:57.685251 16784 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bad62c65be0e436b97992c0c818ebf52 (127.9.166.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: NON_VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: true } } }
I20260621 02:44:57.686012 16649 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bad62c65be0e436b97992c0c818ebf52 (127.9.166.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: NON_VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: true } } }
I20260621 02:44:57.685156 16790 mvcc.cc:205] Tried to move back new op lower bound from 7299112529343143936 to 7299112529279090688. Current Snapshot: MvccSnapshot[applied={T|T < 7299112529343143936}]
I20260621 02:44:57.689234 16793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: NON_VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: true } } }
I20260621 02:44:57.689575 16793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:57.691823 16677 sys_catalog.cc:455] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: NON_VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: true } } }
I20260621 02:44:57.692062 16677 sys_catalog.cc:458] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:44:57.688896 16784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: NON_VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: true } } }
I20260621 02:44:57.695631 16784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:57.697729 16730 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER bad62c65be0e436b97992c0c818ebf52 (127.9.166.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } } }
I20260621 02:44:57.701941 16762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2296b5aeb8f34f529d7886dae3a889cf. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } } }
I20260621 02:44:57.701965 16796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } } }
I20260621 02:44:57.702260 16762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260621 02:44:57.702298 16796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260621 02:44:57.701758 16800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: NON_VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: true } } }
I20260621 02:44:57.702837 16800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260621 02:44:57.705063 16595 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260621 02:44:57.705439 16595 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260621 02:44:57.705626 16595 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: stopping tablet replica
I20260621 02:44:57.723621 16595 master.cc:552] Master@127.9.166.60:35903 shutdown complete.
I20260621 02:44:57.723733 16595 master_runner.cc:305] Clearing existing system tablet
I20260621 02:44:57.724118 16595 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:57.724220 16595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:57.726318 16595 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260621 02:44:57.727752 16802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:57.728070 16595 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:57.728338 16595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260621 02:44:57.728433 16595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/wal
uuid: "9ea8bd55103d4ab7ba775a30111cd5a0"
format_stamp: "Formatted at 2026-06-21 02:44:57 on dist-test-slave-hvgn"
server_key: "eb85fb1b9b9d1c53a5c657f4bca82573"
server_key_iv: "e92f4dd56ab427976d5f8f6a64fa964b"
server_key_version: "encryptionkey@0"
I20260621 02:44:57.731467 16595 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:44:57.736203 16595 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260621 02:44:57.736352 16595 log.cc:1196] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260621 02:44:57.736789 16595 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: Deleting consensus metadata
I20260621 02:44:57.737032 16595 master_runner.cc:315] Copying system tablet from 127.9.166.62:37723
I20260621 02:44:57.739355 16595 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: tablet copy: Beginning tablet copy session from remote peer at address 127.9.166.62:37723
I20260621 02:44:57.753276 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.740134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.9.166.62:37723)
0621 02:44:57.740420 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:57.740430 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:57.740971 (+ 541us) server_negotiation.cc:407] Connection header received
0621 02:44:57.741188 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:57.741194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:57.741267 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:57.741365 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:57.742188 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.743113 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.744894 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.745365 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.745493 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.747756 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:57.747762 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:57.747767 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:57.747820 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:57.750678 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.751294 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.751301 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.751305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.751485 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:57.751859 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:57.751865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:57.751869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:57.752131 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:57.752281 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:57.752672 (+ 391us) server_negotiation.cc:299] Negotiation successful
0621 02:44:57.752902 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260621 02:44:57.753306 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:57.740102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.0.0.1:42426)
0621 02:44:57.740857 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:57.740879 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:57.741035 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:57.741387 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:57.741392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:57.741857 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:57.742041 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:57.742050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.743289 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.743293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:57.744754 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:57.744769 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:57.745562 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:57.745571 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:57.745805 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:57.746455 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:57.746473 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:57.747569 (+ 1096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:57.750810 (+ 3241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:57.750816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:57.750820 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:57.751180 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:57.751594 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:57.751599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:57.751603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:57.751755 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:57.752237 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:57.752242 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:57.752415 (+ 173us) client_negotiation.cc:769] Sending connection context
0621 02:44:57.752549 (+ 134us) client_negotiation.cc:240] Negotiation successful
0621 02:44:57.752870 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":515,"threads_started":1}
I20260621 02:44:57.754313 16056 tablet_copy_service.cc:140] P 2296b5aeb8f34f529d7886dae3a889cf: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9ea8bd55103d4ab7ba775a30111cd5a0 ({username='oryx', principal='oryx/127.9.166.60@KRBTEST.COM'} at 127.0.0.1:42426)
I20260621 02:44:57.754479 16056 tablet_copy_service.cc:161] P 2296b5aeb8f34f529d7886dae3a889cf: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9ea8bd55103d4ab7ba775a30111cd5a0 at {username='oryx', principal='oryx/127.9.166.60@KRBTEST.COM'} at 127.0.0.1:42426: session id = 9ea8bd55103d4ab7ba775a30111cd5a0-00000000000000000000000000000000
I20260621 02:44:57.756783 16056 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf: Tablet Copy: opened 0 blocks and 1 log segments
I20260621 02:44:57.758318 16595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260621 02:44:57.763230 16595 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: tablet copy: Starting download of 0 data blocks...
I20260621 02:44:57.763559 16595 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: tablet copy: Starting download of 1 WAL segments...
I20260621 02:44:57.765691 16595 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260621 02:44:57.768636 16056 tablet_copy_service.cc:342] P 2296b5aeb8f34f529d7886dae3a889cf: Request end of tablet copy session 9ea8bd55103d4ab7ba775a30111cd5a0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.166.60@KRBTEST.COM'} at 127.0.0.1:42426
I20260621 02:44:57.768761 16056 tablet_copy_service.cc:433] P 2296b5aeb8f34f529d7886dae3a889cf: ending tablet copy session 9ea8bd55103d4ab7ba775a30111cd5a0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9ea8bd55103d4ab7ba775a30111cd5a0
I20260621 02:44:57.780117 16595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:57.782526 16809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:57.784529 16810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:57.795305 16812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:57.795706 16595 server_base.cc:1034] running on GCE node
I20260621 02:44:57.796047 16595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:57.796684 16595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:57.797871 16595 hybrid_clock.cc:648] HybridClock initialized: now 1782009897797826 us; error 42 us; skew 500 ppm
I20260621 02:44:57.798133 16595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:57 dist-test-slave-hvgn krb5kdc[15991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009897, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:57.800239 16595 init.cc:377] Logged in from keytab as oryx/127.9.166.60@KRBTEST.COM (short username oryx)
I20260621 02:44:57.823267 16595 webserver.cc:460] Webserver started at http://127.9.166.60:46879/ using document root <none> and password file <none>
I20260621 02:44:57.823675 16595 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:57.823832 16595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:57.824729 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 16097
I20260621 02:44:57.828452 16595 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260621 02:44:57.830446 16819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:57.830874 16595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260621 02:44:57.831009 16595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/wal
uuid: "9ea8bd55103d4ab7ba775a30111cd5a0"
format_stamp: "Formatted at 2026-06-21 02:44:57 on dist-test-slave-hvgn"
server_key: "eb85fb1b9b9d1c53a5c657f4bca82573"
server_key_iv: "e92f4dd56ab427976d5f8f6a64fa964b"
server_key_version: "encryptionkey@0"
I20260621 02:44:57.831193 16595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:57.843269 16595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:57.844605 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:35761
--local_ip_for_outbound_sockets=127.9.166.1
--tserver_master_addrs=127.9.166.62:37723
--webserver_port=46035
--webserver_interface=127.9.166.1
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.1.keytab
--principal=oryx/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.166.62:37723,127.9.166.61:33393
--tserver_master_addrs=127.9.166.62:37723,127.9.166.61:33393,127.9.166.60:35903 with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:44:57.847572 16595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:57.882795 16595 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.60:35903
I20260621 02:44:57.882856 16871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.60:35903 every 8 connection(s)
I20260621 02:44:57.883834 16595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/master-2/data/info.pb
I20260621 02:44:57.888700 16872 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:44:57.892297 16872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: Bootstrap starting.
I20260621 02:44:57.908150 16872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:44:57.908676 16872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: Bootstrap complete.
I20260621 02:44:57.909688 16872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: NON_VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: true } }
I20260621 02:44:57.909951 16872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9ea8bd55103d4ab7ba775a30111cd5a0, State: Initialized, Role: LEARNER
I20260621 02:44:57.910184 16872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: NON_VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: true } }
I20260621 02:44:57.911265 16872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:44:57.911007 16875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: NON_VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: true } } }
I20260621 02:44:57.911623 16875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [sys.catalog]: This master's current role is: LEARNER
I20260621 02:44:57.918545 16886 catalog_manager.cc:1259] Loaded cluster ID: c80c61a5840d48a68a453a9bbfc9acb9
I20260621 02:44:57.918684 16886 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: loading cluster ID for follower catalog manager: success
I20260621 02:44:57.922079 16886 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: acquiring CA information for follower catalog manager: success
I20260621 02:44:57.924362 16886 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 02:44:58.181136 16826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:58.181412 16826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:58.181566 16826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:58.188591 16826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:58.188724 16826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:58.188884 16826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:44:58.198212 16826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.1.keytab
--principal=oryx/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:35761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=46035
--tserver_master_addrs=127.9.166.62:37723,127.9.166.61:33393,127.9.166.60:35903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:58.199430 16826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:58.200656 16826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 02:44:58.213012 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:58.206016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43530 (local address 127.9.166.60:35903)
0621 02:44:58.206698 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:58.206704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:58.206725 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:44:58.206798 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:58.206803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:58.206906 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:58.207061 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:58.207554 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.209065 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.211172 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.212012 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.212169 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:58.212388 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:58.212568 (+ 180us) server_negotiation.cc:299] Negotiation successful
0621 02:44:58.212700 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":353,"threads_started":1}
I20260621 02:44:58.213012 16891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:58.205892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:35903 (local address 127.0.0.1:43530)
0621 02:44:58.206369 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:58.206396 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:58.206550 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:58.207118 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:58.207123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:58.207156 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:58.207362 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:58.207372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.209207 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.209213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:58.211047 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:58.211063 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.212192 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.212199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:58.212388 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:58.212431 (+ 43us) client_negotiation.cc:769] Sending connection context
0621 02:44:58.212548 (+ 117us) client_negotiation.cc:240] Negotiation successful
0621 02:44:58.212688 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":199,"threads_started":1}
I20260621 02:44:58.214349 16847 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20260621 02:44:58.214653 16895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:58.214776 16875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2296b5aeb8f34f529d7886dae3a889cf. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: NON_VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: true } } }
I20260621 02:44:58.215040 16875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [sys.catalog]: This master's current role is: LEARNER
W20260621 02:44:58.215668 16894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:58.215874 16897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:58.217132 16826 server_base.cc:1034] running on GCE node
I20260621 02:44:58.217751 16826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:58.218765 16826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:58.219973 16826 hybrid_clock.cc:648] HybridClock initialized: now 1782009898219908 us; error 49 us; skew 500 ppm
I20260621 02:44:58.220296 16826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:58 dist-test-slave-hvgn krb5kdc[15991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009898, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:58.223757 16826 init.cc:377] Logged in from keytab as oryx/127.9.166.1@KRBTEST.COM (short username oryx)
I20260621 02:44:58.229799 16826 webserver.cc:460] Webserver started at http://127.9.166.1:46035/ using document root <none> and password file <none>
I20260621 02:44:58.230446 16826 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:58.230588 16826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:58.235695 16826 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:44:58.238294 16904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:58.239615 16826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:44:58.239792 16826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "df8a57d499a54ac5bc5d7a5579e910cf"
format_stamp: "Formatted at 2026-06-21 02:44:55 on dist-test-slave-hvgn"
server_key: "b95a64220cb85e192b0337eccb03b309"
server_key_iv: "08d9a441677b0922df4bd2de63f1c573"
server_key_version: "encryptionkey@0"
I20260621 02:44:58.240674 16826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:58.270932 16826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:58.275074 16826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:58.275337 16826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:58.276248 16826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:58.277613 16826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:58.277729 16826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:58.277858 16826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:58.277944 16826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:58.309965 16826 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:35761
I20260621 02:44:58.310043 17016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:35761 every 8 connection(s)
I20260621 02:44:58.311461 16826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:44:58.317883 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 16826
I20260621 02:44:58.318264 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 16231
Jun 21 02:44:58 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009898, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM
Jun 21 02:44:58 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009898, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.61@KRBTEST.COM
Jun 21 02:44:58 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.166.2@KRBTEST.COM
Jun 21 02:44:58 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009898, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.60@KRBTEST.COM
I20260621 02:44:58.363236 17022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:58.314346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:58627 (local address 127.9.166.62:37723)
0621 02:44:58.337862 (+ 23516us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:58.337867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:58.338477 (+ 610us) server_negotiation.cc:407] Connection header received
0621 02:44:58.342110 (+ 3633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:58.342116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:58.342191 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:58.342305 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:58.345149 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.346341 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.349434 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.350067 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.350165 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:58.355289 (+ 5124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:58.355297 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:58.355306 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:58.355351 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:58.359668 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:58.360508 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:58.360513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:58.360517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:58.360639 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:58.361146 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:58.361152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:58.361155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:58.361480 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:58.361603 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:58.362539 (+ 936us) server_negotiation.cc:299] Negotiation successful
0621 02:44:58.362781 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23384,"thread_start_us":23511,"threads_started":1}
I20260621 02:44:58.363597 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:58.315298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:44345 (local address 127.9.166.60:35903)
0621 02:44:58.315580 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:58.315588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:58.342080 (+ 26492us) server_negotiation.cc:407] Connection header received
0621 02:44:58.342320 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:58.342327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:58.342417 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:58.342494 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:58.345201 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.346287 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.349452 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.350145 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.350258 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:58.356837 (+ 6579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:58.356845 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:58.356850 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:58.356890 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:58.360454 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:58.361335 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:58.361340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:58.361343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:58.361481 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:58.361953 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:58.361961 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:58.361967 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:58.362322 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:58.362446 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:58.363016 (+ 570us) server_negotiation.cc:299] Negotiation successful
0621 02:44:58.363222 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260621 02:44:58.365132 17027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:58.319771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36504 (local address 127.9.166.2:39959)
0621 02:44:58.337432 (+ 17661us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:58.337439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:58.340639 (+ 3200us) server_negotiation.cc:407] Connection header received
0621 02:44:58.340879 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:58.340889 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:58.341160 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:58.341323 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:58.344277 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.346120 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.348229 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.349069 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.349168 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:58.356858 (+ 7690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:58.356871 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:58.356885 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:58.356921 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:58.361149 (+ 4228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:58.362010 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:58.362020 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:58.362023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:58.362151 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:58.362676 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:58.362681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:58.362684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:58.363199 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:58.363937 (+ 738us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:58.364525 (+ 588us) server_negotiation.cc:299] Negotiation successful
0621 02:44:58.364776 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17515,"thread_start_us":17649,"threads_started":1}
I20260621 02:44:58.365676 17023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:58.314889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:35903 (local address 127.9.166.1:44345)
0621 02:44:58.341975 (+ 27086us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:58.341994 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:58.342172 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:58.342731 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:58.342743 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:58.343734 (+ 991us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:58.345028 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:58.345043 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.346532 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.346539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:58.349260 (+ 2721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:58.349275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.350544 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.350550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:58.350918 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:58.351979 (+ 1061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:58.352006 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:58.356602 (+ 4596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:58.360605 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:58.360609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:58.360613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:58.361147 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:58.361628 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:58.361632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:58.361636 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:58.361820 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:58.362525 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:58.362530 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:58.362708 (+ 178us) client_negotiation.cc:769] Sending connection context
0621 02:44:58.363777 (+ 1069us) client_negotiation.cc:240] Negotiation successful
0621 02:44:58.364201 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25704,"mutex_wait_us":706,"spinlock_wait_cycles":62720,"thread_start_us":23302,"threads_started":1}
I20260621 02:44:58.365705 17025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:58.314889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.1:58627)
0621 02:44:58.338270 (+ 23381us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:58.338338 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:58.341674 (+ 3336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:58.342729 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:58.342741 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:58.343740 (+ 999us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:58.344997 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:58.345011 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.346535 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.346540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:58.349276 (+ 2736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:58.349289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.350508 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.350518 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:58.350921 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:58.351841 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:58.351880 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:58.355003 (+ 3123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:58.359813 (+ 4810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:58.359823 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:58.359838 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:58.360378 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:58.360767 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:58.360771 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:58.360775 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:58.361012 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:58.361716 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:58.361724 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:58.362083 (+ 359us) client_negotiation.cc:769] Sending connection context
0621 02:44:58.363535 (+ 1452us) client_negotiation.cc:240] Negotiation successful
0621 02:44:58.364349 (+ 814us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22956,"mutex_wait_us":828,"spinlock_wait_cycles":49024,"thread_start_us":26664,"threads_started":1}
I20260621 02:44:58.366665 17021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:58.314239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:45413 (local address 127.9.166.61:33393)
0621 02:44:58.337025 (+ 22786us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:58.337033 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:58.341236 (+ 4203us) server_negotiation.cc:407] Connection header received
0621 02:44:58.342073 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:58.342078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:58.342154 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:58.342232 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:58.345119 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.346338 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.349434 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.350083 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.350165 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:58.355686 (+ 5521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:58.355693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:58.355698 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:58.355748 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:58.359569 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:58.364382 (+ 4813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:58.364387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:58.364391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:58.364532 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:58.365070 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:58.365075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:58.365078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:58.365409 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:58.365552 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:58.366104 (+ 552us) server_negotiation.cc:299] Negotiation successful
0621 02:44:58.366324 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22649,"thread_start_us":22767,"threads_started":1}
I20260621 02:44:58.367534 17024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:58.315893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:33393 (local address 127.9.166.1:45413)
0621 02:44:58.341130 (+ 25237us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:58.341148 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:58.341918 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:58.342729 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:58.342741 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:58.344011 (+ 1270us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:58.344903 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:58.344929 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.346632 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.346639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:58.349277 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:58.349289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.350746 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.350752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:58.351034 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:58.351841 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:58.351879 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:58.355484 (+ 3605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:58.363602 (+ 8118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:58.363606 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:58.363610 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:58.364245 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:58.364676 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:58.364680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:58.364684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:58.364894 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:58.365636 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:58.365645 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:58.365811 (+ 166us) client_negotiation.cc:769] Sending connection context
0621 02:44:58.366848 (+ 1037us) client_negotiation.cc:240] Negotiation successful
0621 02:44:58.367127 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25113,"mutex_wait_us":593,"spinlock_wait_cycles":2432,"thread_start_us":25409,"threads_started":1}
I20260621 02:44:58.367648 17017 heartbeater.cc:347] Connected to a master server at 127.9.166.60:35903
I20260621 02:44:58.367945 17017 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:58.367960 16354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:58.368381 17019 heartbeater.cc:347] Connected to a master server at 127.9.166.61:33393
I20260621 02:44:58.368507 17019 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:58.368929 17017 heartbeater.cc:511] Master 127.9.166.60:35903 requested a full tablet report, sending...
I20260621 02:44:58.369733 17018 heartbeater.cc:347] Connected to a master server at 127.9.166.62:37723
I20260621 02:44:58.369853 17018 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:58.370253 17018 heartbeater.cc:511] Master 127.9.166.62:37723 requested a full tablet report, sending...
I20260621 02:44:58.368970 17019 heartbeater.cc:511] Master 127.9.166.61:33393 requested a full tablet report, sending...
I20260621 02:44:58.371807 16035 ts_manager.cc:194] Re-registered known tserver with Master: df8a57d499a54ac5bc5d7a5579e910cf (127.9.166.1:35761)
I20260621 02:44:58.372285 16639 ts_manager.cc:194] Registered new tserver with Master: df8a57d499a54ac5bc5d7a5579e910cf (127.9.166.1:35761)
I20260621 02:44:58.372526 16837 ts_manager.cc:194] Registered new tserver with Master: df8a57d499a54ac5bc5d7a5579e910cf (127.9.166.1:35761)
I20260621 02:44:58.372948 16035 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:58627
I20260621 02:44:58.455644 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 16231
I20260621 02:44:58.473006 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:39959
--local_ip_for_outbound_sockets=127.9.166.2
--tserver_master_addrs=127.9.166.62:37723
--webserver_port=46465
--webserver_interface=127.9.166.2
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.2.keytab
--principal=oryx/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.166.62:37723,127.9.166.61:33393
--tserver_master_addrs=127.9.166.62:37723,127.9.166.61:33393,127.9.166.60:35903 with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:44:58.681952 16890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: NON_VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: true } } }
I20260621 02:44:58.682416 16890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:58.683756 16890 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf: attempting to promote NON_VOTER 9ea8bd55103d4ab7ba775a30111cd5a0 to VOTER
I20260621 02:44:58.684432 16890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: false } }
I20260621 02:44:58.686458 16847 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 1 LEARNER]: Refusing update from remote peer 2296b5aeb8f34f529d7886dae3a889cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260621 02:44:58.686682 16649 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 1 FOLLOWER]: Refusing update from remote peer 2296b5aeb8f34f529d7886dae3a889cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260621 02:44:58.687011 17031 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260621 02:44:58.687513 16890 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260621 02:44:58.690295 16649 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9ea8bd55103d4ab7ba775a30111cd5a0 (127.9.166.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: false } } }
I20260621 02:44:58.690968 16847 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9ea8bd55103d4ab7ba775a30111cd5a0 (127.9.166.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: false } } }
I20260621 02:44:58.689718 16890 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9ea8bd55103d4ab7ba775a30111cd5a0 (127.9.166.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: false } } }
I20260621 02:44:58.694986 16875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: false } } }
I20260621 02:44:58.695268 16875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:44:58.696516 17031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: false } } }
I20260621 02:44:58.696642 16890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: false } } }
I20260621 02:44:58.696834 16890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:58.696838 17031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:58.697101 16890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: false } } }
I20260621 02:44:58.697253 16890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf [sys.catalog]: This master's current role is: LEADER
I20260621 02:44:58.697772 17034 sys_catalog.cc:455] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2296b5aeb8f34f529d7886dae3a889cf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: false } } }
I20260621 02:44:58.698130 17034 sys_catalog.cc:458] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: This master's current role is: FOLLOWER
W20260621 02:44:58.794185 17029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:58.794461 17029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:58.794622 17029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:58.801689 17029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:58.801819 17029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:58.801997 17029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:44:58.816210 17029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.2.keytab
--principal=oryx/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:39959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=46465
--tserver_master_addrs=127.9.166.62:37723,127.9.166.61:33393,127.9.166.60:35903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:58.817765 17029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:58.819015 17029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:58.832252 17039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:58.832301 17040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:58.832327 17042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:58.833546 17029 server_base.cc:1034] running on GCE node
I20260621 02:44:58.834113 17029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:58.834978 17029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:58.836166 17029 hybrid_clock.cc:648] HybridClock initialized: now 1782009898836127 us; error 36 us; skew 500 ppm
I20260621 02:44:58.836438 17029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:58 dist-test-slave-hvgn krb5kdc[15991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009898, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:58.839493 17029 init.cc:377] Logged in from keytab as oryx/127.9.166.2@KRBTEST.COM (short username oryx)
I20260621 02:44:58.840903 17029 webserver.cc:460] Webserver started at http://127.9.166.2:46465/ using document root <none> and password file <none>
I20260621 02:44:58.841612 17029 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:58.841748 17029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:58.847759 17029 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 02:44:58.850359 17049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:58.851960 17029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:44:58.852092 17029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "57c34184a77044d9ae5d7b754069936b"
format_stamp: "Formatted at 2026-06-21 02:44:55 on dist-test-slave-hvgn"
server_key: "8fc357722cbfa5d371f349240b0ffe74"
server_key_iv: "3ce8f05447a90ad2057d667b0a9db145"
server_key_version: "encryptionkey@0"
I20260621 02:44:58.853044 17029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:58.872852 17029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:58.877081 17029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:58.877307 17029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:58.878211 17029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:58.879629 17029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:58.879712 17029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:58.879792 17029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:58.879851 17029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:58.913151 17029 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:39959
I20260621 02:44:58.913327 17161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:39959 every 8 connection(s)
I20260621 02:44:58.914736 17029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:44:58.921705 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 17029
I20260621 02:44:58.922075 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 16364
Jun 21 02:44:58 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009898, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.61@KRBTEST.COM
Jun 21 02:44:58 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009898, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.60@KRBTEST.COM
Jun 21 02:44:58 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.166.3@KRBTEST.COM
Jun 21 02:44:58 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009898, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM
I20260621 02:44:58.962057 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:58.919150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:38171 (local address 127.9.166.61:33393)
0621 02:44:58.930339 (+ 11189us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:58.930347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:58.930373 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:44:58.930466 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:58.930474 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:58.930571 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:58.930690 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:58.938890 (+ 8200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.940265 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.946154 (+ 5889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.946810 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.946920 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:58.953443 (+ 6523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:58.953450 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:58.953456 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:58.953508 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:58.957417 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:58.959194 (+ 1777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:58.959200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:58.959203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:58.959344 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:58.959922 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:58.959926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:58.959929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:58.960325 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:58.960456 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:58.961380 (+ 924us) server_negotiation.cc:299] Negotiation successful
0621 02:44:58.961610 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11037,"thread_start_us":10932,"threads_started":1}
I20260621 02:44:58.964025 17166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:58.917632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:57279 (local address 127.9.166.60:35903)
0621 02:44:58.934721 (+ 17089us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:58.934730 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:58.937378 (+ 2648us) server_negotiation.cc:407] Connection header received
0621 02:44:58.937460 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:58.937468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:58.937580 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:58.937695 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:58.942339 (+ 4644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.943677 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.946192 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.946892 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.947021 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:58.953372 (+ 6351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:58.953379 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:58.953384 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:58.953434 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:58.957990 (+ 4556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:58.959139 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:58.959145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:58.959149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:58.959290 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:58.959864 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:58.959869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:58.959872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:58.960236 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:58.960379 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:58.963374 (+ 2995us) server_negotiation.cc:299] Negotiation successful
0621 02:44:58.963633 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16910,"thread_start_us":17488,"threads_started":1}
I20260621 02:44:58.964126 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:58.918723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:39241 (local address 127.9.166.62:37723)
0621 02:44:58.935108 (+ 16385us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:58.935117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:58.938653 (+ 3536us) server_negotiation.cc:407] Connection header received
0621 02:44:58.938737 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:58.938745 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:58.938853 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:58.938970 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:58.944767 (+ 5797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.946115 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.948108 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.948610 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.948742 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:58.955014 (+ 6272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:58.955021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:58.955025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:58.955066 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:58.960329 (+ 5263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:58.961155 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:58.961161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:58.961164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:58.961320 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:58.961737 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:58.961742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:58.961745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:58.962076 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:58.962212 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:58.963508 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0621 02:44:58.963726 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16238,"thread_start_us":16131,"threads_started":1}
I20260621 02:44:58.964327 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:58.923716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41854 (local address 127.9.166.3:46685)
0621 02:44:58.927559 (+ 3843us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:58.927571 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:58.936553 (+ 8982us) server_negotiation.cc:407] Connection header received
0621 02:44:58.939617 (+ 3064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:58.939628 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:58.939877 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:58.940019 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:58.941874 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.943952 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.946168 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.947157 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.947281 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:58.955350 (+ 8069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:58.955365 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:58.955406 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:58.955452 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:58.960452 (+ 5000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:58.961275 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:58.961357 (+ 82us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:58.961364 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:58.961566 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:58.962022 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:58.962030 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:58.962034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:58.962431 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:58.962723 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:58.963597 (+ 874us) server_negotiation.cc:299] Negotiation successful
0621 02:44:58.963852 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3604,"thread_start_us":3436,"threads_started":1}
I20260621 02:44:58.964543 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:58.918389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:33393 (local address 127.9.166.2:38171)
0621 02:44:58.926763 (+ 8374us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:58.926820 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:58.929602 (+ 2782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:58.935600 (+ 5998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:58.935616 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:58.936717 (+ 1101us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:58.938566 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:58.938595 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.942102 (+ 3507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.942108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:58.945987 (+ 3879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:58.946001 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.947313 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.947324 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:58.947718 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:58.948620 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:58.948666 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:58.952230 (+ 3564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:58.958490 (+ 6260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:58.958498 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:58.958503 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:58.959048 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:58.959526 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:58.959530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:58.959534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:58.959785 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:58.960549 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:58.960553 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:58.960952 (+ 399us) client_negotiation.cc:769] Sending connection context
0621 02:44:58.962502 (+ 1550us) client_negotiation.cc:240] Negotiation successful
0621 02:44:58.962789 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8008,"mutex_wait_us":46,"spinlock_wait_cycles":41344,"thread_start_us":9353,"threads_started":1}
I20260621 02:44:58.964622 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:58.918389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.2:39241)
0621 02:44:58.938341 (+ 19952us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:58.938363 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:58.938521 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:58.939167 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:58.939174 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:58.944220 (+ 5046us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:58.944591 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:58.944606 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.946265 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.946270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:58.947923 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:58.947939 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.948999 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.949005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:58.949247 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:58.949836 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:58.949857 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:58.954771 (+ 4914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:58.960456 (+ 5685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:58.960460 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:58.960463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:58.961000 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:58.961447 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:58.961452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:58.961455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:58.961608 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:58.962270 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:58.962275 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:58.962544 (+ 269us) client_negotiation.cc:769] Sending connection context
0621 02:44:58.962691 (+ 147us) client_negotiation.cc:240] Negotiation successful
0621 02:44:58.963010 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19771,"mutex_wait_us":2440,"thread_start_us":15821,"threads_started":1}
I20260621 02:44:58.964593 17171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:58.919648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:35903 (local address 127.9.166.2:57279)
0621 02:44:58.937064 (+ 17416us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:58.937087 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:58.937249 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:58.938523 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:58.938530 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:58.939178 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:58.942185 (+ 3007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:58.942199 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.943850 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.943856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:58.945946 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:58.945960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:58.947312 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:58.947323 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:58.947709 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:58.948684 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:58.948715 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:58.953170 (+ 4455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:58.958142 (+ 4972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:58.958151 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:58.958168 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:58.958984 (+ 816us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:58.959454 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:58.959467 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:58.959471 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:58.959708 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:58.960452 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:58.960472 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:58.960935 (+ 463us) client_negotiation.cc:769] Sending connection context
0621 02:44:58.961238 (+ 303us) client_negotiation.cc:240] Negotiation successful
0621 02:44:58.961787 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17251,"mutex_wait_us":79,"thread_start_us":18333,"threads_started":1}
I20260621 02:44:58.965736 16487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:44:58.966084 17163 heartbeater.cc:347] Connected to a master server at 127.9.166.62:37723
I20260621 02:44:58.966373 17163 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:58.966852 17162 heartbeater.cc:347] Connected to a master server at 127.9.166.60:35903
I20260621 02:44:58.967012 17162 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:58.967303 17163 heartbeater.cc:511] Master 127.9.166.62:37723 requested a full tablet report, sending...
I20260621 02:44:58.967646 17162 heartbeater.cc:511] Master 127.9.166.60:35903 requested a full tablet report, sending...
I20260621 02:44:58.968124 17164 heartbeater.cc:347] Connected to a master server at 127.9.166.61:33393
I20260621 02:44:58.968258 17164 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:58.968648 17164 heartbeater.cc:511] Master 127.9.166.61:33393 requested a full tablet report, sending...
I20260621 02:44:58.969163 16837 ts_manager.cc:194] Registered new tserver with Master: 57c34184a77044d9ae5d7b754069936b (127.9.166.2:39959)
I20260621 02:44:58.969432 16035 ts_manager.cc:194] Re-registered known tserver with Master: 57c34184a77044d9ae5d7b754069936b (127.9.166.2:39959)
I20260621 02:44:58.969928 16639 ts_manager.cc:194] Registered new tserver with Master: 57c34184a77044d9ae5d7b754069936b (127.9.166.2:39959)
I20260621 02:44:58.970362 16035 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:39241
I20260621 02:44:59.053874 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 16364
I20260621 02:44:59.070824 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:46685
--local_ip_for_outbound_sockets=127.9.166.3
--tserver_master_addrs=127.9.166.62:37723
--webserver_port=40615
--webserver_interface=127.9.166.3
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.3.keytab
--principal=oryx/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.166.62:37723,127.9.166.61:33393
--tserver_master_addrs=127.9.166.62:37723,127.9.166.61:33393,127.9.166.60:35903 with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:44:59.374996 17018 heartbeater.cc:503] Master 127.9.166.62:37723 was elected leader, sending a full tablet report...
W20260621 02:44:59.390357 17175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:44:59.390610 17175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:44:59.390777 17175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:44:59.397938 17175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:44:59.398032 17175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:44:59.398195 17175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:44:59.407374 17175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.3.keytab
--principal=oryx/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:46685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=40615
--tserver_master_addrs=127.9.166.62:37723,127.9.166.61:33393,127.9.166.60:35903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:44:59.408598 17175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:44:59.409904 17175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:44:59.423079 17181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:59.423120 17180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:44:59.423152 17183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:44:59.424238 17175 server_base.cc:1034] running on GCE node
I20260621 02:44:59.424890 17175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:44:59.425763 17175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:44:59.426985 17175 hybrid_clock.cc:648] HybridClock initialized: now 1782009899426948 us; error 29 us; skew 500 ppm
I20260621 02:44:59.427340 17175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:44:59 dist-test-slave-hvgn krb5kdc[15991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009899, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:44:59.430449 17175 init.cc:377] Logged in from keytab as oryx/127.9.166.3@KRBTEST.COM (short username oryx)
I20260621 02:44:59.431864 17175 webserver.cc:460] Webserver started at http://127.9.166.3:40615/ using document root <none> and password file <none>
I20260621 02:44:59.432566 17175 fs_manager.cc:359] Metadata directory not provided
I20260621 02:44:59.432714 17175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:44:59.437829 17175 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 02:44:59.440603 17190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:59.442440 17175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:44:59.442605 17175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "b220186e43e34781af1b15d2d60be62c"
format_stamp: "Formatted at 2026-06-21 02:44:56 on dist-test-slave-hvgn"
server_key: "354e7622d6b7c07057b38f34b3741a37"
server_key_iv: "f6d7f84dbb4779206d995a8ecba2dcbc"
server_key_version: "encryptionkey@0"
I20260621 02:44:59.444160 17175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:44:59.460824 17175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:44:59.465691 17175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:44:59.465965 17175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:44:59.471817 17175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:44:59.473613 17175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:44:59.473739 17175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:59.473855 17175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:44:59.473941 17175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:44:59.506151 17175 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:46685
I20260621 02:44:59.506173 17302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:46685 every 8 connection(s)
I20260621 02:44:59.508051 17175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:44:59.509950 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 17175
Jun 21 02:44:59 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009899, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.60@KRBTEST.COM
Jun 21 02:44:59 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009899, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.61@KRBTEST.COM
I20260621 02:44:59.572528 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:59.526740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:38845 (local address 127.9.166.60:35903)
0621 02:44:59.540652 (+ 13912us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:59.540661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:59.550764 (+ 10103us) server_negotiation.cc:407] Connection header received
0621 02:44:59.551032 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:59.551040 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:59.551128 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:59.551206 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:59.552541 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.553587 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.555732 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.556417 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.556522 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:59.561936 (+ 5414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:59.561942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:59.561947 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:59.561984 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:59.565978 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:59.567072 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:59.567081 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:59.567087 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:59.567257 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:59.567891 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:59.567900 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:59.567906 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:59.568359 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:59.568528 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:59.571614 (+ 3086us) server_negotiation.cc:299] Negotiation successful
0621 02:44:59.571951 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13762,"thread_start_us":13648,"threads_started":1}
I20260621 02:44:59.573387 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:59.526749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:35903 (local address 127.9.166.3:38845)
0621 02:44:59.550643 (+ 23894us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:59.550663 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:59.550882 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:59.551447 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:59.551454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:59.552047 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:59.552375 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:59.552386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.553769 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.553775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:59.555520 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:59.555593 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.556975 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.556984 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:59.557403 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:59.558303 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:59.558333 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:59.561537 (+ 3204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:59.566108 (+ 4571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:59.566116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:59.566134 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:59.566891 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:59.567426 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:59.567434 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:59.567440 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:59.567707 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:59.568624 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:59.568637 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:59.569176 (+ 539us) client_negotiation.cc:769] Sending connection context
0621 02:44:59.569537 (+ 361us) client_negotiation.cc:240] Negotiation successful
0621 02:44:59.570206 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23752,"mutex_wait_us":76,"thread_start_us":18450,"threads_started":1}
I20260621 02:44:59.575129 17303 heartbeater.cc:347] Connected to a master server at 127.9.166.60:35903
I20260621 02:44:59.575500 17303 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:59.575593 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:59.512508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:38929 (local address 127.9.166.61:33393)
0621 02:44:59.555575 (+ 43067us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:59.555581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:59.555601 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:44:59.555656 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:59.555661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:59.555750 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:59.557024 (+ 1274us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:59.557210 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.558796 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.561045 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.561638 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.561760 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:59.566617 (+ 4857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:59.566627 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:59.566635 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:59.566704 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:59.571246 (+ 4542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:59.572127 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:59.572132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:59.572135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:59.572289 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:59.573092 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:59.573100 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:59.573106 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:59.573792 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:59.573958 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:59.574543 (+ 585us) server_negotiation.cc:299] Negotiation successful
0621 02:44:59.574895 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42938,"thread_start_us":42846,"threads_started":1}
I20260621 02:44:59.576447 17303 heartbeater.cc:511] Master 127.9.166.60:35903 requested a full tablet report, sending...
I20260621 02:44:59.576851 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:59.512478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:33393 (local address 127.9.166.3:38929)
0621 02:44:59.544889 (+ 32411us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:59.544915 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:59.545112 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:59.555913 (+ 10801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:59.555918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:59.556468 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:59.556880 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:59.556891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.559039 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.559044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:59.560834 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:59.560853 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.561880 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.561890 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:59.562154 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:59.563427 (+ 1273us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:59.563458 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:59.566334 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:59.571438 (+ 5104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:59.571445 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:59.571450 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:59.571978 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:59.572455 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:59.572462 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:59.572468 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:59.572951 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:59.574047 (+ 1096us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:59.574055 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:59.574228 (+ 173us) client_negotiation.cc:769] Sending connection context
0621 02:44:59.575990 (+ 1762us) client_negotiation.cc:240] Negotiation successful
0621 02:44:59.576306 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32010,"thread_start_us":38570,"threads_started":1}
I20260621 02:44:59.578311 17305 heartbeater.cc:347] Connected to a master server at 127.9.166.61:33393
I20260621 02:44:59.578455 17305 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:59.578481 16837 ts_manager.cc:194] Registered new tserver with Master: b220186e43e34781af1b15d2d60be62c (127.9.166.3:46685)
I20260621 02:44:59.578826 17305 heartbeater.cc:511] Master 127.9.166.61:33393 requested a full tablet report, sending...
I20260621 02:44:59.579977 16639 ts_manager.cc:194] Registered new tserver with Master: b220186e43e34781af1b15d2d60be62c (127.9.166.3:46685)
Jun 21 02:44:59 dist-test-slave-hvgn krb5kdc[15991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009899, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM
I20260621 02:44:59.587869 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:59.511678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.3:57795)
0621 02:44:59.536807 (+ 25129us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:59.536879 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:59.543115 (+ 6236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:59.548640 (+ 5525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:59.548679 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:59.549917 (+ 1238us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:44:59.550938 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:59.550965 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.552732 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.552737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:59.561728 (+ 8991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:59.561740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.562785 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.562795 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:59.563036 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:59.564853 (+ 1817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:44:59.564875 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:44:59.580932 (+ 16057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:44:59.585101 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:59.585107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:59.585111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:59.585438 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:59.586102 (+ 664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:44:59.586108 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:44:59.586111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:44:59.586244 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:44:59.586805 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:44:59.586810 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:44:59.586953 (+ 143us) client_negotiation.cc:769] Sending connection context
0621 02:44:59.587096 (+ 143us) client_negotiation.cc:240] Negotiation successful
0621 02:44:59.587373 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24740,"mutex_wait_us":6704,"thread_start_us":27576,"threads_started":1}
I20260621 02:44:59.588321 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:59.511758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:57795 (local address 127.9.166.62:37723)
0621 02:44:59.547843 (+ 36085us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:59.547852 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:59.547878 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:44:59.547959 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:59.547967 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:59.548068 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:59.548187 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:59.551333 (+ 3146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.552535 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.561851 (+ 9316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.562538 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.562678 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:59.581117 (+ 18439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:59.581124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:59.581128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:59.581170 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:59.584987 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:59.585567 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:59.585572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:59.585575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:59.585995 (+ 420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:59.586357 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:59.586362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:59.586365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:59.586682 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:59.586816 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:59.587204 (+ 388us) server_negotiation.cc:299] Negotiation successful
0621 02:44:59.587452 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35929,"thread_start_us":36537,"threads_started":1}
I20260621 02:44:59.589289 17304 heartbeater.cc:347] Connected to a master server at 127.9.166.62:37723
I20260621 02:44:59.589439 17304 heartbeater.cc:464] Registering TS with master...
I20260621 02:44:59.589882 17304 heartbeater.cc:511] Master 127.9.166.62:37723 requested a full tablet report, sending...
I20260621 02:44:59.590865 16035 ts_manager.cc:194] Re-registered known tserver with Master: b220186e43e34781af1b15d2d60be62c (127.9.166.3:46685)
I20260621 02:44:59.592093 16035 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:57795
I20260621 02:44:59.596088 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:59.566289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42442 (local address 127.9.166.62:37723)
0621 02:44:59.576642 (+ 10353us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:59.576648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:59.576667 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:44:59.576722 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:59.576727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:59.576797 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:59.580661 (+ 3864us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:59.580856 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.582184 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.584228 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.584971 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.585086 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:59.588863 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:59.588871 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:59.588877 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:59.588922 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:59.592431 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:59.593122 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:59.593127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:59.593131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:59.593267 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:59.593623 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:59.593628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:59.593631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:59.593872 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:59.594108 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:59.595424 (+ 1316us) server_negotiation.cc:299] Negotiation successful
0621 02:44:59.595655 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10223,"thread_start_us":13199,"threads_started":1}
I20260621 02:44:59.597836 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:59.572099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43538 (local address 127.9.166.60:35903)
0621 02:44:59.573924 (+ 1825us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:59.573929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:59.577517 (+ 3588us) server_negotiation.cc:407] Connection header received
0621 02:44:59.580634 (+ 3117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:59.580639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:59.580741 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:59.580849 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:59.583045 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.584095 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.585999 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.586659 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.586767 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:59.590388 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:59.590394 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:59.590399 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:59.590433 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:59.594544 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:59.595646 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:59.595653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:59.595659 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:59.595838 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:59.596215 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:59.596220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:59.596224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:59.596572 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:59.596823 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:59.597203 (+ 380us) server_negotiation.cc:299] Negotiation successful
0621 02:44:59.597476 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1695,"thread_start_us":3436,"threads_started":1}
I20260621 02:44:59.602890 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:59.577403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32936 (local address 127.9.166.61:33393)
0621 02:44:59.577581 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:59.577589 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:59.580087 (+ 2498us) server_negotiation.cc:407] Connection header received
0621 02:44:59.581274 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:59.581278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:59.581365 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:59.581434 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:44:59.583835 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.584904 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.590250 (+ 5346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.590941 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.591049 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:59.596528 (+ 5479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:44:59.596534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:44:59.596539 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:44:59.596573 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:44:59.600162 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:59.600754 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:59.600761 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:59.600764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:59.600921 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:44:59.601234 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:44:59.601239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:44:59.601242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:44:59.601566 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:44:59.601794 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:59.602026 (+ 232us) server_negotiation.cc:299] Negotiation successful
0621 02:44:59.602552 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 02:44:59.600548 16035 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:44:59.608451 16035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:44:59.635982 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:59.623986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:35761 (local address 127.0.0.1:45580)
0621 02:44:59.624548 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:59.624568 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:59.624707 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:59.625850 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:59.625855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:59.625883 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:59.626305 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:59.626335 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.629184 (+ 2849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.629193 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:59.632064 (+ 2871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:59.632083 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.634882 (+ 2799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.634892 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:59.635143 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:59.635201 (+ 58us) client_negotiation.cc:769] Sending connection context
0621 02:44:59.635289 (+ 88us) client_negotiation.cc:240] Negotiation successful
0621 02:44:59.635492 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":359,"threads_started":1}
I20260621 02:44:59.637706 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:59.624524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.9.166.1:35761)
0621 02:44:59.625033 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:59.625042 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:59.625068 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:44:59.625172 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:59.625183 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:59.625566 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:59.626573 (+ 1007us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:59.626899 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.628967 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.632345 (+ 3378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.634573 (+ 2228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.636640 (+ 2067us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:59.636960 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:59.637062 (+ 102us) server_negotiation.cc:299] Negotiation successful
0621 02:44:59.637300 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":676,"threads_started":1}
I20260621 02:44:59.642249 16952 tablet_service.cc:1467] Processing CreateTablet for tablet 474b3e905f9d4978bc6e3ec6b6f729be (DEFAULT_TABLE table=test-table [id=922a7fc95eb442de9044eff301ff520a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:59.643807 16952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474b3e905f9d4978bc6e3ec6b6f729be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:59.646488 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:59.625142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:46685 (local address 127.0.0.1:41860)
0621 02:44:59.636216 (+ 11074us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:59.636232 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:59.636319 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:59.639408 (+ 3089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:59.639417 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:59.639456 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:59.639685 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:59.639699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.642145 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.642150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:59.644090 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:59.644104 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.645717 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.645723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:59.645901 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:59.645937 (+ 36us) client_negotiation.cc:769] Sending connection context
0621 02:44:59.646026 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 02:44:59.646141 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10999,"mutex_wait_us":1,"thread_start_us":16103,"threads_started":1}
I20260621 02:44:59.647321 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:59.625426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41860 (local address 127.9.166.3:46685)
0621 02:44:59.638422 (+ 12996us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:59.638435 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:59.638477 (+ 42us) server_negotiation.cc:407] Connection header received
0621 02:44:59.638597 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:59.638609 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:59.638956 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:59.639921 (+ 965us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:59.640247 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.642017 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.644237 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.645551 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.645733 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:59.645934 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:59.646767 (+ 833us) server_negotiation.cc:299] Negotiation successful
0621 02:44:59.646937 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12766,"thread_start_us":13680,"threads_started":1}
I20260621 02:44:59.649461 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:59.624704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:39959 (local address 127.0.0.1:36506)
0621 02:44:59.631524 (+ 6820us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:59.631540 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:59.631667 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:59.642389 (+ 10722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:59.642393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:59.642424 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:59.642793 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:59.642802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.644487 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.644492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:59.646424 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:59.646440 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.648534 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.648540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:59.648711 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:59.648782 (+ 71us) client_negotiation.cc:769] Sending connection context
0621 02:44:59.648882 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 02:44:59.649006 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6662,"mutex_wait_us":23,"thread_start_us":6515,"threads_started":1}
I20260621 02:44:59.650141 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:59.625069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.9.166.2:39959)
0621 02:44:59.641725 (+ 16656us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:59.641733 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:59.641818 (+ 85us) server_negotiation.cc:407] Connection header received
0621 02:44:59.641925 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:59.641934 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:59.642185 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:59.642400 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:59.643067 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.644325 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.646736 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.648344 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.648472 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:59.648654 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:59.648961 (+ 307us) server_negotiation.cc:299] Negotiation successful
0621 02:44:59.649110 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16515,"thread_start_us":16416,"threads_started":1}
I20260621 02:44:59.650961 17238 tablet_service.cc:1467] Processing CreateTablet for tablet 474b3e905f9d4978bc6e3ec6b6f729be (DEFAULT_TABLE table=test-table [id=922a7fc95eb442de9044eff301ff520a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:59.652356 17238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474b3e905f9d4978bc6e3ec6b6f729be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:59.654268 17097 tablet_service.cc:1467] Processing CreateTablet for tablet 474b3e905f9d4978bc6e3ec6b6f729be (DEFAULT_TABLE table=test-table [id=922a7fc95eb442de9044eff301ff520a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:44:59.655493 17097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474b3e905f9d4978bc6e3ec6b6f729be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:44:59.658982 17336 tablet_bootstrap.cc:492] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf: Bootstrap starting.
I20260621 02:44:59.662585 17336 tablet_bootstrap.cc:654] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:59.663834 17336 log.cc:824] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:59.666616 17338 tablet_bootstrap.cc:492] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c: Bootstrap starting.
I20260621 02:44:59.666836 17336 tablet_bootstrap.cc:492] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf: No bootstrap required, opened a new log
I20260621 02:44:59.667315 17336 ts_tablet_manager.cc:1397] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260621 02:44:59.669602 17339 tablet_bootstrap.cc:492] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b: Bootstrap starting.
I20260621 02:44:59.670217 17338 tablet_bootstrap.cc:654] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:59.671465 17338 log.cc:824] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:59.672760 17339 tablet_bootstrap.cc:654] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b: Neither blocks nor log segments found. Creating new log.
I20260621 02:44:59.673899 17339 log.cc:824] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b: Log is configured to *not* fsync() on all Append() calls
I20260621 02:44:59.674456 17338 tablet_bootstrap.cc:492] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c: No bootstrap required, opened a new log
I20260621 02:44:59.674675 17338 ts_tablet_manager.cc:1397] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260621 02:44:59.676198 17336 raft_consensus.cc:348] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 } } peers { permanent_uuid: "57c34184a77044d9ae5d7b754069936b" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39959 } } peers { permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 } }
I20260621 02:44:59.676613 17336 raft_consensus.cc:374] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:59.676721 17336 raft_consensus.cc:729] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8a57d499a54ac5bc5d7a5579e910cf, State: Initialized, Role: FOLLOWER
I20260621 02:44:59.677101 17339 tablet_bootstrap.cc:492] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b: No bootstrap required, opened a new log
I20260621 02:44:59.677364 17339 ts_tablet_manager.cc:1397] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260621 02:44:59.677335 17336 consensus_queue.cc:260] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 } } peers { permanent_uuid: "57c34184a77044d9ae5d7b754069936b" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39959 } } peers { permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 } }
I20260621 02:44:59.679059 17336 ts_tablet_manager.cc:1428] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260621 02:44:59.684614 17338 raft_consensus.cc:348] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 } } peers { permanent_uuid: "57c34184a77044d9ae5d7b754069936b" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39959 } } peers { permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 } }
I20260621 02:44:59.685041 17338 raft_consensus.cc:374] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:59.685135 17338 raft_consensus.cc:729] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b220186e43e34781af1b15d2d60be62c, State: Initialized, Role: FOLLOWER
I20260621 02:44:59.685720 17338 consensus_queue.cc:260] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 } } peers { permanent_uuid: "57c34184a77044d9ae5d7b754069936b" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39959 } } peers { permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 } }
I20260621 02:44:59.687347 17338 ts_tablet_manager.cc:1428] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260621 02:44:59.687636 17304 heartbeater.cc:503] Master 127.9.166.62:37723 was elected leader, sending a full tablet report...
I20260621 02:44:59.690552 17339 raft_consensus.cc:348] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 } } peers { permanent_uuid: "57c34184a77044d9ae5d7b754069936b" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39959 } } peers { permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 } }
I20260621 02:44:59.690966 17339 raft_consensus.cc:374] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:44:59.691073 17339 raft_consensus.cc:729] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57c34184a77044d9ae5d7b754069936b, State: Initialized, Role: FOLLOWER
I20260621 02:44:59.691682 17339 consensus_queue.cc:260] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 } } peers { permanent_uuid: "57c34184a77044d9ae5d7b754069936b" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39959 } } peers { permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 } }
I20260621 02:44:59.692580 17163 heartbeater.cc:503] Master 127.9.166.62:37723 was elected leader, sending a full tablet report...
I20260621 02:44:59.693131 17339 ts_tablet_manager.cc:1428] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b: Time spent starting tablet: real 0.016s user 0.007s sys 0.005s
W20260621 02:44:59.694617 17165 tablet.cc:2369] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:44:59.811501 17306 tablet.cc:2369] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:44:59.814947 17020 tablet.cc:2369] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:44:59.968127 17344 raft_consensus.cc:482] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:44:59.968401 17344 raft_consensus.cc:504] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 } } peers { permanent_uuid: "57c34184a77044d9ae5d7b754069936b" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39959 } } peers { permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 } }
I20260621 02:44:59.969985 17344 leader_election.cc:288] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df8a57d499a54ac5bc5d7a5579e910cf (127.9.166.1:35761), b220186e43e34781af1b15d2d60be62c (127.9.166.3:46685)
I20260621 02:44:59.976619 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:59.970682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:46787 (local address 127.9.166.3:46685)
0621 02:44:59.970888 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:59.970895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:59.971436 (+ 541us) server_negotiation.cc:407] Connection header received
0621 02:44:59.971678 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:59.971682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:59.971764 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:59.971849 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:59.972435 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.973083 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.974870 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.975656 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.975750 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:59.975880 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:59.976224 (+ 344us) server_negotiation.cc:299] Negotiation successful
0621 02:44:59.976349 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 02:44:59.976627 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:59.970355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:60389 (local address 127.9.166.1:35761)
0621 02:44:59.970589 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 02:44:59.970599 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:44:59.971352 (+ 753us) server_negotiation.cc:407] Connection header received
0621 02:44:59.971574 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:44:59.971583 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:44:59.971679 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:44:59.971747 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:44:59.972362 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.973087 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.974880 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.975691 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.975845 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:59.975938 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 02:44:59.976250 (+ 312us) server_negotiation.cc:299] Negotiation successful
0621 02:44:59.976380 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:44:59.976820 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:59.970254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:35761 (local address 127.9.166.2:60389)
0621 02:44:59.971223 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:59.971247 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:59.971379 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:59.971793 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:59.971800 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:59.971829 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:59.972209 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:59.972217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.973220 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.973225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:59.974731 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:59.974743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.975937 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.975942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:59.976078 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:59.976110 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 02:44:59.976215 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 02:44:59.976349 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":31,"spinlock_wait_cycles":896,"thread_start_us":532,"threads_started":1}
I20260621 02:44:59.977109 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:44:59.970253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:46685 (local address 127.9.166.2:46787)
0621 02:44:59.971335 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0621 02:44:59.971352 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:44:59.971538 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:44:59.971895 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:44:59.971899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:44:59.971919 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:44:59.972266 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:59.972303 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.973213 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.973221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:59.974736 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:44:59.974747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:44:59.975859 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:44:59.975865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:44:59.976066 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:44:59.976101 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 02:44:59.976175 (+ 74us) client_negotiation.cc:240] Negotiation successful
0621 02:44:59.976312 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"mutex_wait_us":196,"thread_start_us":695,"threads_started":1}
I20260621 02:44:59.978142 16972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474b3e905f9d4978bc6e3ec6b6f729be" candidate_uuid: "57c34184a77044d9ae5d7b754069936b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" is_pre_election: true
I20260621 02:44:59.978636 16972 raft_consensus.cc:2454] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57c34184a77044d9ae5d7b754069936b in term 0.
I20260621 02:44:59.979254 17050 leader_election.cc:302] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57c34184a77044d9ae5d7b754069936b, df8a57d499a54ac5bc5d7a5579e910cf; no voters:
I20260621 02:44:59.979425 17258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474b3e905f9d4978bc6e3ec6b6f729be" candidate_uuid: "57c34184a77044d9ae5d7b754069936b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b220186e43e34781af1b15d2d60be62c" is_pre_election: true
I20260621 02:44:59.979750 17344 raft_consensus.cc:2781] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:44:59.979894 17344 raft_consensus.cc:482] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:44:59.979883 17258 raft_consensus.cc:2454] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57c34184a77044d9ae5d7b754069936b in term 0.
I20260621 02:44:59.980002 17344 raft_consensus.cc:3037] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:59.982256 17344 raft_consensus.cc:504] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 } } peers { permanent_uuid: "57c34184a77044d9ae5d7b754069936b" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39959 } } peers { permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 } }
I20260621 02:44:59.982890 17344 leader_election.cc:288] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [CANDIDATE]: Term 1 election: Requested vote from peers df8a57d499a54ac5bc5d7a5579e910cf (127.9.166.1:35761), b220186e43e34781af1b15d2d60be62c (127.9.166.3:46685)
I20260621 02:44:59.983213 16972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474b3e905f9d4978bc6e3ec6b6f729be" candidate_uuid: "57c34184a77044d9ae5d7b754069936b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8a57d499a54ac5bc5d7a5579e910cf"
I20260621 02:44:59.983282 17258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474b3e905f9d4978bc6e3ec6b6f729be" candidate_uuid: "57c34184a77044d9ae5d7b754069936b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b220186e43e34781af1b15d2d60be62c"
I20260621 02:44:59.983443 16972 raft_consensus.cc:3037] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:59.983564 17258 raft_consensus.cc:3037] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:44:59.985544 16972 raft_consensus.cc:2454] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57c34184a77044d9ae5d7b754069936b in term 1.
I20260621 02:44:59.985683 17258 raft_consensus.cc:2454] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57c34184a77044d9ae5d7b754069936b in term 1.
I20260621 02:44:59.985882 17050 leader_election.cc:302] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57c34184a77044d9ae5d7b754069936b, df8a57d499a54ac5bc5d7a5579e910cf; no voters:
I20260621 02:44:59.986174 17344 raft_consensus.cc:2781] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:44:59.986598 17344 raft_consensus.cc:686] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [term 1 LEADER]: Becoming Leader. State: Replica: 57c34184a77044d9ae5d7b754069936b, State: Running, Role: LEADER
I20260621 02:44:59.987341 17344 consensus_queue.cc:237] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 } } peers { permanent_uuid: "57c34184a77044d9ae5d7b754069936b" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39959 } } peers { permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 } }
I20260621 02:44:59.992117 16035 catalog_manager.cc:5495] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b reported cstate change: term changed from 0 to 1, leader changed from <none> to 57c34184a77044d9ae5d7b754069936b (127.9.166.2). New cstate: current_term: 1 leader_uuid: "57c34184a77044d9ae5d7b754069936b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57c34184a77044d9ae5d7b754069936b" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 } health_report { overall_health: UNKNOWN } } }
I20260621 02:45:00.010803 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:00.005760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36514 (local address 127.9.166.2:39959)
0621 02:45:00.005991 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:00.005995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:00.006012 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:45:00.006185 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:00.006188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:00.006268 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:00.006421 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:45:00.006734 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:00.007486 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:00.008826 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:00.009212 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:00.009305 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:00.009721 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:45:00.010243 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:45:00.010357 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:00.010441 (+ 84us) server_negotiation.cc:299] Negotiation successful
0621 02:45:00.010499 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 02:45:00.018714 17258 raft_consensus.cc:1261] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c [term 1 FOLLOWER]: Refusing update from remote peer 57c34184a77044d9ae5d7b754069936b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:45:00.018846 16972 raft_consensus.cc:1261] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf [term 1 FOLLOWER]: Refusing update from remote peer 57c34184a77044d9ae5d7b754069936b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:45:00.019691 17347 consensus_queue.cc:1035] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:45:00.019994 17344 consensus_queue.cc:1035] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:45:00.032917 17351 mvcc.cc:205] Tried to move back new op lower bound from 7299112550467715072 to 7299112550351671296. Current Snapshot: MvccSnapshot[applied={T|T < 7299112550467715072}]
I20260621 02:45:00.054373 16035 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:45:00.054623 16035 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:45:00.060571 16035 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2296b5aeb8f34f529d7886dae3a889cf: Sending DeleteTablet for 3 replicas of tablet 474b3e905f9d4978bc6e3ec6b6f729be
I20260621 02:45:00.061805 16952 tablet_service.cc:1514] Processing DeleteTablet for tablet 474b3e905f9d4978bc6e3ec6b6f729be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:00 UTC) from {username='oryx', principal='oryx/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:45580
I20260621 02:45:00.061908 17097 tablet_service.cc:1514] Processing DeleteTablet for tablet 474b3e905f9d4978bc6e3ec6b6f729be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:00 UTC) from {username='oryx', principal='oryx/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:36506
I20260621 02:45:00.064482 17238 tablet_service.cc:1514] Processing DeleteTablet for tablet 474b3e905f9d4978bc6e3ec6b6f729be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:00 UTC) from {username='oryx', principal='oryx/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:41860
I20260621 02:45:00.088279 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 16004
I20260621 02:45:00.088277 17359 tablet_replica.cc:330] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf: stopping tablet replica
I20260621 02:45:00.091044 16066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:00.091369 17359 raft_consensus.cc:2229] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:45:00.094058 17359 raft_consensus.cc:2258] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:00.095314 17360 tablet_replica.cc:330] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b: stopping tablet replica
I20260621 02:45:00.095847 17360 raft_consensus.cc:2229] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:45:00.098567 17359 ts_tablet_manager.cc:1905] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:45:00.107177 17359 ts_tablet_manager.cc:1918] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:45:00.107409 17360 raft_consensus.cc:2258] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:00.107707 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:00.063685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.9.166.61:33393)
0621 02:45:00.063888 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:00.063897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:00.086826 (+ 22929us) server_negotiation.cc:407] Connection header received
0621 02:45:00.087055 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:00.087065 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:00.087173 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:00.087306 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:00.093327 (+ 6021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:00.094812 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:00.096469 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:00.097096 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:00.097235 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:00.100810 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:00.100817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:00.100824 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:00.100866 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:00.104374 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:00.105175 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:00.105180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:00.105184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:00.105325 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:00.105771 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:00.105776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:00.105779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:00.106153 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:00.106391 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:00.106927 (+ 536us) server_negotiation.cc:299] Negotiation successful
0621 02:45:00.107163 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260621 02:45:00.111268 17362 tablet_replica.cc:330] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c: stopping tablet replica
I20260621 02:45:00.112365 17362 raft_consensus.cc:2229] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:45:00.112915 17362 raft_consensus.cc:2258] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:00.113713 17360 ts_tablet_manager.cc:1905] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:45:00.120052 17362 ts_tablet_manager.cc:1905] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:45:00.107353 17359 log.cc:1196] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/474b3e905f9d4978bc6e3ec6b6f729be
I20260621 02:45:00.128001 17359 ts_tablet_manager.cc:1939] T 474b3e905f9d4978bc6e3ec6b6f729be P df8a57d499a54ac5bc5d7a5579e910cf: Deleting consensus metadata
I20260621 02:45:00.128494 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:00.087639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43546 (local address 127.9.166.60:35903)
0621 02:45:00.087810 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:00.087816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:00.109095 (+ 21279us) server_negotiation.cc:407] Connection header received
0621 02:45:00.109363 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:00.109369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:00.109450 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:00.109518 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:00.111604 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:00.112762 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:00.115314 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:00.115995 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:00.116092 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:00.121273 (+ 5181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:00.121280 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:00.121285 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:00.121332 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:00.124907 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:00.125960 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:00.125965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:00.125968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:00.126103 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:00.126591 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:00.126596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:00.126599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:00.126953 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:00.127184 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:00.127831 (+ 647us) server_negotiation.cc:299] Negotiation successful
0621 02:45:00.128045 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 02:45:00.132364 17360 ts_tablet_manager.cc:1918] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:45:00.132536 17360 log.cc:1196] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/474b3e905f9d4978bc6e3ec6b6f729be
I20260621 02:45:00.133172 17360 ts_tablet_manager.cc:1939] T 474b3e905f9d4978bc6e3ec6b6f729be P 57c34184a77044d9ae5d7b754069936b: Deleting consensus metadata
I20260621 02:45:00.134938 17362 ts_tablet_manager.cc:1918] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:45:00.135107 17362 log.cc:1196] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/474b3e905f9d4978bc6e3ec6b6f729be
I20260621 02:45:00.135655 17362 ts_tablet_manager.cc:1939] T 474b3e905f9d4978bc6e3ec6b6f729be P b220186e43e34781af1b15d2d60be62c: Deleting consensus metadata
I20260621 02:45:00.251063 16020 catalog_manager.cc:4841] TS 57c34184a77044d9ae5d7b754069936b (127.9.166.2:39959): tablet 474b3e905f9d4978bc6e3ec6b6f729be (table test-table [id=922a7fc95eb442de9044eff301ff520a]) successfully deleted
I20260621 02:45:00.251443 16021 catalog_manager.cc:4841] TS b220186e43e34781af1b15d2d60be62c (127.9.166.3:46685): tablet 474b3e905f9d4978bc6e3ec6b6f729be (table test-table [id=922a7fc95eb442de9044eff301ff520a]) successfully deleted
I20260621 02:45:00.251726 16020 catalog_manager.cc:4841] TS df8a57d499a54ac5bc5d7a5579e910cf (127.9.166.1:35761): tablet 474b3e905f9d4978bc6e3ec6b6f729be (table test-table [id=922a7fc95eb442de9044eff301ff520a]) successfully deleted
I20260621 02:45:00.251816 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 16004
I20260621 02:45:00.305693 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:00.277939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43552 (local address 127.9.166.60:35903)
0621 02:45:00.278138 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:00.278146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:00.288380 (+ 10234us) server_negotiation.cc:407] Connection header received
0621 02:45:00.288935 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:00.288945 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:00.289052 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:00.289141 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:00.292833 (+ 3692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:00.294059 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:00.295991 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:00.296631 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:00.296755 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:00.300407 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:00.300415 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:00.300423 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:00.300466 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:00.303035 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:00.303697 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:00.303701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:00.303704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:00.303811 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:00.304186 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:00.304190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:00.304194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:00.304458 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:00.304705 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:00.304991 (+ 286us) server_negotiation.cc:299] Negotiation successful
0621 02:45:00.305340 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 02:45:00.306865 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:00.277768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.9.166.61:33393)
0621 02:45:00.278017 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:00.278023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:00.292245 (+ 14222us) server_negotiation.cc:407] Connection header received
0621 02:45:00.292504 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:00.292511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:00.292594 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:00.292661 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:00.294903 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:00.295624 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:00.297346 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:00.297916 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:00.298050 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:00.300480 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:00.300487 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:00.300492 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:00.300532 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:00.303832 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:00.304643 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:00.304648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:00.304654 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:00.304826 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:00.305321 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:00.305329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:00.305335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:00.305665 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:00.306242 (+ 577us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:00.306360 (+ 118us) server_negotiation.cc:299] Negotiation successful
0621 02:45:00.306557 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260621 02:45:01.047204 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.046506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.3:34235)
0621 02:45:01.046907 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.047014 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":197,"threads_started":1}
W20260621 02:45:01.047689 17304 heartbeater.cc:644] Failed to heartbeat to 127.9.166.62:37723 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
I20260621 02:45:01.048904 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.047965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.1:41723)
0621 02:45:01.048568 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.048706 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":389,"threads_started":1}
W20260621 02:45:01.049399 17018 heartbeater.cc:644] Failed to heartbeat to 127.9.166.62:37723 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
I20260621 02:45:01.052304 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.051675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.2:34957)
0621 02:45:01.052078 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.052172 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":183,"threads_started":1}
W20260621 02:45:01.052644 17163 heartbeater.cc:644] Failed to heartbeat to 127.9.166.62:37723 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
I20260621 02:45:01.561789 17375 raft_consensus.cc:482] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2296b5aeb8f34f529d7886dae3a889cf)
I20260621 02:45:01.561759 17374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2296b5aeb8f34f529d7886dae3a889cf)
I20260621 02:45:01.562009 17374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: false } }
I20260621 02:45:01.562053 17375 raft_consensus.cc:504] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: false } }
I20260621 02:45:01.562980 17374 leader_election.cc:288] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2296b5aeb8f34f529d7886dae3a889cf (127.9.166.62:37723), bad62c65be0e436b97992c0c818ebf52 (127.9.166.61:33393)
I20260621 02:45:01.565248 17375 leader_election.cc:288] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2296b5aeb8f34f529d7886dae3a889cf (127.9.166.62:37723), 9ea8bd55103d4ab7ba775a30111cd5a0 (127.9.166.60:35903)
I20260621 02:45:01.572765 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.563737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.0.0.1:42838)
0621 02:45:01.572124 (+ 8387us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.572356 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8214,"threads_started":1}
W20260621 02:45:01.573330 16624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
I20260621 02:45:01.575958 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.563146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.0.0.1:42830)
0621 02:45:01.575470 (+ 12324us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.575695 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12139,"thread_start_us":11959,"threads_started":1}
W20260621 02:45:01.576213 16821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
I20260621 02:45:01.577872 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.576866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.0.0.1:42844)
0621 02:45:01.577585 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.577703 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":597}
W20260621 02:45:01.578073 16624 leader_election.cc:334] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2296b5aeb8f34f529d7886dae3a889cf (127.9.166.62:37723): Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
I20260621 02:45:01.579872 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.579458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.0.0.1:42852)
0621 02:45:01.579596 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.579696 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260621 02:45:01.580884 16821 leader_election.cc:334] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2296b5aeb8f34f529d7886dae3a889cf (127.9.166.62:37723): Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
I20260621 02:45:01.583321 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.563146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:33393 (local address 127.0.0.1:32972)
0621 02:45:01.575675 (+ 12529us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.575694 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:01.575847 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:01.576897 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:01.576905 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:01.576941 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:45:01.577381 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:01.577397 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.579316 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.579323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:01.581453 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:01.581466 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.582592 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.582603 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:01.582731 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:01.582757 (+ 26us) client_negotiation.cc:769] Sending connection context
0621 02:45:01.582834 (+ 77us) client_negotiation.cc:240] Negotiation successful
0621 02:45:01.583015 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12401,"mutex_wait_us":172,"thread_start_us":12164,"threads_started":1}
I20260621 02:45:01.583344 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.563459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32972 (local address 127.9.166.61:33393)
0621 02:45:01.576337 (+ 12878us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:01.576345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:01.576366 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:45:01.576438 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:01.576442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:01.576513 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:01.576620 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:45:01.577710 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.579150 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.581606 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.582435 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.582558 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:01.582660 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:01.582870 (+ 210us) server_negotiation.cc:299] Negotiation successful
0621 02:45:01.583019 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":12733,"thread_start_us":12548,"threads_started":1}
I20260621 02:45:01.583987 16649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bad62c65be0e436b97992c0c818ebf52" is_pre_election: true
I20260621 02:45:01.584312 16649 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ea8bd55103d4ab7ba775a30111cd5a0 in term 1.
I20260621 02:45:01.584756 16823 leader_election.cc:302] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ea8bd55103d4ab7ba775a30111cd5a0, bad62c65be0e436b97992c0c818ebf52; no voters: 2296b5aeb8f34f529d7886dae3a889cf
I20260621 02:45:01.584975 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.577337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:35903 (local address 127.0.0.1:43568)
0621 02:45:01.578007 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.578030 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:01.578290 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:01.579023 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:01.579028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:01.579052 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:45:01.579513 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:01.579527 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.580863 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.580868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:01.582438 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:01.582449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.584213 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.584219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:01.584415 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:01.584451 (+ 36us) client_negotiation.cc:769] Sending connection context
0621 02:45:01.584561 (+ 110us) client_negotiation.cc:240] Negotiation successful
0621 02:45:01.584685 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":1,"thread_start_us":1006,"threads_started":1}
I20260621 02:45:01.585117 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.576559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43568 (local address 127.9.166.60:35903)
0621 02:45:01.577442 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:01.577451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:01.578114 (+ 663us) server_negotiation.cc:407] Connection header received
0621 02:45:01.578466 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:01.578474 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:01.578544 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:01.578628 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:45:01.579712 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.580739 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.582604 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.584005 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.584129 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:01.584223 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:01.584602 (+ 379us) server_negotiation.cc:299] Negotiation successful
0621 02:45:01.584732 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":732,"thread_start_us":582,"threads_started":1}
I20260621 02:45:01.585124 17374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260621 02:45:01.585291 17374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2296b5aeb8f34f529d7886dae3a889cf)
I20260621 02:45:01.585383 17374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 1 FOLLOWER]: Advancing to term 2
I20260621 02:45:01.587494 17374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: false } }
I20260621 02:45:01.588109 16847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bad62c65be0e436b97992c0c818ebf52" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" is_pre_election: true
I20260621 02:45:01.588287 17374 leader_election.cc:288] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [CANDIDATE]: Term 2 election: Requested vote from peers 2296b5aeb8f34f529d7886dae3a889cf (127.9.166.62:37723), bad62c65be0e436b97992c0c818ebf52 (127.9.166.61:33393)
I20260621 02:45:01.588518 16847 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bad62c65be0e436b97992c0c818ebf52 in current term 2: Already voted for candidate 9ea8bd55103d4ab7ba775a30111cd5a0 in this term.
I20260621 02:45:01.588712 16649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bad62c65be0e436b97992c0c818ebf52"
I20260621 02:45:01.588932 16649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 1 FOLLOWER]: Advancing to term 2
I20260621 02:45:01.588929 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.588502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.0.0.1:42864)
0621 02:45:01.588663 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.588753 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260621 02:45:01.589329 16625 leader_election.cc:302] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bad62c65be0e436b97992c0c818ebf52; no voters: 2296b5aeb8f34f529d7886dae3a889cf, 9ea8bd55103d4ab7ba775a30111cd5a0
I20260621 02:45:01.590672 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.590320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.0.0.1:42866)
0621 02:45:01.590472 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.590556 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260621 02:45:01.590842 16821 leader_election.cc:334] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2296b5aeb8f34f529d7886dae3a889cf (127.9.166.62:37723): Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
I20260621 02:45:01.591562 16649 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ea8bd55103d4ab7ba775a30111cd5a0 in term 2.
I20260621 02:45:01.591831 17375 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260621 02:45:01.592020 16823 leader_election.cc:302] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ea8bd55103d4ab7ba775a30111cd5a0, bad62c65be0e436b97992c0c818ebf52; no voters: 2296b5aeb8f34f529d7886dae3a889cf
I20260621 02:45:01.592288 17374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 02:45:01.592738 17374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [term 2 LEADER]: Becoming Leader. State: Replica: 9ea8bd55103d4ab7ba775a30111cd5a0, State: Running, Role: LEADER
I20260621 02:45:01.593099 17374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: false } }
I20260621 02:45:01.594597 17384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ea8bd55103d4ab7ba775a30111cd5a0. Latest consensus state: current_term: 2 leader_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: false } } }
I20260621 02:45:01.594920 17384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [sys.catalog]: This master's current role is: LEADER
I20260621 02:45:01.595640 17386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:45:01.598986 17386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:45:01.600513 17386 catalog_manager.cc:1259] Loaded cluster ID: c80c61a5840d48a68a453a9bbfc9acb9
I20260621 02:45:01.600581 17386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:45:01.601663 17386 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:45:01.602455 17386 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: Loaded TSK: 0
I20260621 02:45:01.603245 17386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:45:01.656612 16836 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:45:01.664435 16836 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:45:01.668704 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.668310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.0.0.1:42896)
0621 02:45:01.668476 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.668562 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260621 02:45:01.668833 16649 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [term 2 FOLLOWER]: Refusing update from remote peer 9ea8bd55103d4ab7ba775a30111cd5a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260621 02:45:01.669579 17384 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260621 02:45:01.669824 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.669398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.0.0.1:42912)
0621 02:45:01.669589 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.669659 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 02:45:01.669992 16821 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 -> Peer 2296b5aeb8f34f529d7886dae3a889cf (127.9.166.62:37723): Couldn't send request to peer 2296b5aeb8f34f529d7886dae3a889cf. Status: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260621 02:45:01.672212 17375 sys_catalog.cc:455] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ea8bd55103d4ab7ba775a30111cd5a0. Latest consensus state: current_term: 2 leader_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: false } } }
I20260621 02:45:01.672466 17375 sys_catalog.cc:458] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:45:01.674008 17374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: false } } }
I20260621 02:45:01.674425 17374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0 [sys.catalog]: This master's current role is: LEADER
I20260621 02:45:01.674741 17375 sys_catalog.cc:455] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2296b5aeb8f34f529d7886dae3a889cf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37723 } } peers { permanent_uuid: "bad62c65be0e436b97992c0c818ebf52" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 33393 } attrs { promote: false } } peers { permanent_uuid: "9ea8bd55103d4ab7ba775a30111cd5a0" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 35903 } attrs { promote: false } } }
I20260621 02:45:01.675040 17375 sys_catalog.cc:458] T 00000000000000000000000000000000 P bad62c65be0e436b97992c0c818ebf52 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:45:01.693030 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.682262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:39959 (local address 127.0.0.1:36526)
0621 02:45:01.683151 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.683169 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:01.683327 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:01.683786 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:01.683794 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:01.683828 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:45:01.684232 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:01.684248 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.686392 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.686401 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:01.689256 (+ 2855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:01.689276 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.691960 (+ 2684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.691969 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:01.692253 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:01.692311 (+ 58us) client_negotiation.cc:769] Sending connection context
0621 02:45:01.692407 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 02:45:01.692620 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776}
I20260621 02:45:01.694669 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.682377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.9.166.2:39959)
0621 02:45:01.682837 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:01.682843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:01.683262 (+ 419us) server_negotiation.cc:407] Connection header received
0621 02:45:01.683503 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:01.683509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:01.683610 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:01.684517 (+ 907us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:45:01.684775 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.686184 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.689900 (+ 3716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.691682 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.694102 (+ 2420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:01.694230 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:01.694283 (+ 53us) server_negotiation.cc:299] Negotiation successful
0621 02:45:01.694439 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":1084,"threads_started":1}
I20260621 02:45:01.695449 17097 tablet_service.cc:1467] Processing CreateTablet for tablet 53314f3399c142f1af6a3ab3558ac15a (DEFAULT_TABLE table=test-table [id=476e76d0f06e4ad6b7f46c545f552cea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:45:01.695933 17097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53314f3399c142f1af6a3ab3558ac15a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:01.699651 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.682742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:35761 (local address 127.0.0.1:45596)
0621 02:45:01.689598 (+ 6856us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.689623 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:01.691279 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:01.694149 (+ 2870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:01.694156 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:01.694179 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:45:01.694361 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:01.694373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.695991 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.695996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:01.697434 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:01.697446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.698876 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.698883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:01.699071 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:01.699095 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 02:45:01.699206 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 02:45:01.699331 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6708}
I20260621 02:45:01.699676 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.682866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45596 (local address 127.9.166.1:35761)
0621 02:45:01.693787 (+ 10921us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:01.693796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:01.693820 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:45:01.693905 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:01.693914 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:01.694024 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:01.694122 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:45:01.694600 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.695859 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.697576 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.698720 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.698854 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:01.698954 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:01.699231 (+ 277us) server_negotiation.cc:299] Negotiation successful
0621 02:45:01.699358 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10782,"thread_start_us":10689,"threads_started":1}
I20260621 02:45:01.700311 16952 tablet_service.cc:1467] Processing CreateTablet for tablet 53314f3399c142f1af6a3ab3558ac15a (DEFAULT_TABLE table=test-table [id=476e76d0f06e4ad6b7f46c545f552cea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:45:01.700635 16952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53314f3399c142f1af6a3ab3558ac15a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:01.701901 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.683052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:46685 (local address 127.0.0.1:41876)
0621 02:45:01.693363 (+ 10311us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.693386 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:01.693501 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:01.696146 (+ 2645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:01.696153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:01.696186 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:45:01.696350 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:01.696364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.697759 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.697765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:01.699645 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:01.699662 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.701200 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.701205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:01.701410 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:01.701442 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 02:45:01.701504 (+ 62us) client_negotiation.cc:240] Negotiation successful
0621 02:45:01.701633 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10175,"thread_start_us":11123,"threads_started":1}
I20260621 02:45:01.702713 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.683279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41876 (local address 127.9.166.3:46685)
0621 02:45:01.695778 (+ 12499us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:01.695785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:01.695806 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:45:01.695883 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:01.695888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:01.695969 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:01.696119 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:45:01.696516 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.697599 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.699855 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.701036 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.702113 (+ 1077us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:01.702203 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:01.702251 (+ 48us) server_negotiation.cc:299] Negotiation successful
0621 02:45:01.702417 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12372,"thread_start_us":12266,"threads_started":1}
I20260621 02:45:01.702740 17396 tablet_bootstrap.cc:492] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b: Bootstrap starting.
I20260621 02:45:01.704125 17238 tablet_service.cc:1467] Processing CreateTablet for tablet 53314f3399c142f1af6a3ab3558ac15a (DEFAULT_TABLE table=test-table [id=476e76d0f06e4ad6b7f46c545f552cea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:45:01.704520 17238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53314f3399c142f1af6a3ab3558ac15a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:01.704600 17396 tablet_bootstrap.cc:654] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:01.706766 17396 tablet_bootstrap.cc:492] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b: No bootstrap required, opened a new log
I20260621 02:45:01.706916 17396 ts_tablet_manager.cc:1397] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260621 02:45:01.708000 17396 raft_consensus.cc:348] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c34184a77044d9ae5d7b754069936b" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39959 } } peers { permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 } } peers { permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 } }
I20260621 02:45:01.708313 17396 raft_consensus.cc:374] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:01.708436 17396 raft_consensus.cc:729] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57c34184a77044d9ae5d7b754069936b, State: Initialized, Role: FOLLOWER
I20260621 02:45:01.708742 17396 consensus_queue.cc:260] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c34184a77044d9ae5d7b754069936b" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39959 } } peers { permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 } } peers { permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 } }
I20260621 02:45:01.709571 17398 tablet_bootstrap.cc:492] T 53314f3399c142f1af6a3ab3558ac15a P df8a57d499a54ac5bc5d7a5579e910cf: Bootstrap starting.
I20260621 02:45:01.711259 17398 tablet_bootstrap.cc:654] T 53314f3399c142f1af6a3ab3558ac15a P df8a57d499a54ac5bc5d7a5579e910cf: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:01.712095 17396 ts_tablet_manager.cc:1428] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260621 02:45:01.712124 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.710811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.2:41963)
0621 02:45:01.711500 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.711697 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"threads_started":1}
I20260621 02:45:01.714339 17398 tablet_bootstrap.cc:492] T 53314f3399c142f1af6a3ab3558ac15a P df8a57d499a54ac5bc5d7a5579e910cf: No bootstrap required, opened a new log
I20260621 02:45:01.714531 17398 ts_tablet_manager.cc:1397] T 53314f3399c142f1af6a3ab3558ac15a P df8a57d499a54ac5bc5d7a5579e910cf: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260621 02:45:01.715472 17162 heartbeater.cc:503] Master 127.9.166.60:35903 was elected leader, sending a full tablet report...
I20260621 02:45:01.715611 17398 raft_consensus.cc:348] T 53314f3399c142f1af6a3ab3558ac15a P df8a57d499a54ac5bc5d7a5579e910cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c34184a77044d9ae5d7b754069936b" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39959 } } peers { permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 } } peers { permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 } }
I20260621 02:45:01.715941 17398 raft_consensus.cc:374] T 53314f3399c142f1af6a3ab3558ac15a P df8a57d499a54ac5bc5d7a5579e910cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:01.716051 17398 raft_consensus.cc:729] T 53314f3399c142f1af6a3ab3558ac15a P df8a57d499a54ac5bc5d7a5579e910cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8a57d499a54ac5bc5d7a5579e910cf, State: Initialized, Role: FOLLOWER
I20260621 02:45:01.716418 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.716003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.2:45401)
0621 02:45:01.716195 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.716287 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260621 02:45:01.716346 17398 consensus_queue.cc:260] T 53314f3399c142f1af6a3ab3558ac15a P df8a57d499a54ac5bc5d7a5579e910cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c34184a77044d9ae5d7b754069936b" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39959 } } peers { permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 } } peers { permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 } }
I20260621 02:45:01.717528 17398 ts_tablet_manager.cc:1428] T 53314f3399c142f1af6a3ab3558ac15a P df8a57d499a54ac5bc5d7a5579e910cf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260621 02:45:01.719198 17017 heartbeater.cc:503] Master 127.9.166.60:35903 was elected leader, sending a full tablet report...
I20260621 02:45:01.719645 17400 tablet_bootstrap.cc:492] T 53314f3399c142f1af6a3ab3558ac15a P b220186e43e34781af1b15d2d60be62c: Bootstrap starting.
I20260621 02:45:01.721949 17400 tablet_bootstrap.cc:654] T 53314f3399c142f1af6a3ab3558ac15a P b220186e43e34781af1b15d2d60be62c: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:01.725131 17400 tablet_bootstrap.cc:492] T 53314f3399c142f1af6a3ab3558ac15a P b220186e43e34781af1b15d2d60be62c: No bootstrap required, opened a new log
I20260621 02:45:01.725297 17400 ts_tablet_manager.cc:1397] T 53314f3399c142f1af6a3ab3558ac15a P b220186e43e34781af1b15d2d60be62c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 02:45:01.726308 17400 raft_consensus.cc:348] T 53314f3399c142f1af6a3ab3558ac15a P b220186e43e34781af1b15d2d60be62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c34184a77044d9ae5d7b754069936b" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39959 } } peers { permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 } } peers { permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 } }
I20260621 02:45:01.726603 17400 raft_consensus.cc:374] T 53314f3399c142f1af6a3ab3558ac15a P b220186e43e34781af1b15d2d60be62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:01.726715 17400 raft_consensus.cc:729] T 53314f3399c142f1af6a3ab3558ac15a P b220186e43e34781af1b15d2d60be62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b220186e43e34781af1b15d2d60be62c, State: Initialized, Role: FOLLOWER
I20260621 02:45:01.727012 17400 consensus_queue.cc:260] T 53314f3399c142f1af6a3ab3558ac15a P b220186e43e34781af1b15d2d60be62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c34184a77044d9ae5d7b754069936b" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39959 } } peers { permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 } } peers { permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 } }
I20260621 02:45:01.727913 17400 ts_tablet_manager.cc:1428] T 53314f3399c142f1af6a3ab3558ac15a P b220186e43e34781af1b15d2d60be62c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260621 02:45:01.728338 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.718111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.1:46825)
0621 02:45:01.727859 (+ 9748us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.728061 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9601,"thread_start_us":9465,"threads_started":1}
I20260621 02:45:01.729627 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.728203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.3:55715)
0621 02:45:01.729234 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.729342 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":894,"thread_start_us":774,"threads_started":1}
I20260621 02:45:01.729703 17303 heartbeater.cc:503] Master 127.9.166.60:35903 was elected leader, sending a full tablet report...
I20260621 02:45:01.730175 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.729847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.1:59673)
0621 02:45:01.729982 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.730052 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260621 02:45:01.730943 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.730561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.3:39949)
0621 02:45:01.730716 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.730794 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260621 02:45:01.779695 17399 raft_consensus.cc:482] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:45:01.779978 17399 raft_consensus.cc:504] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c34184a77044d9ae5d7b754069936b" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39959 } } peers { permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 } } peers { permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 } }
I20260621 02:45:01.780736 17399 leader_election.cc:288] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df8a57d499a54ac5bc5d7a5579e910cf (127.9.166.1:35761), b220186e43e34781af1b15d2d60be62c (127.9.166.3:46685)
I20260621 02:45:01.781247 16972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53314f3399c142f1af6a3ab3558ac15a" candidate_uuid: "57c34184a77044d9ae5d7b754069936b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" is_pre_election: true
I20260621 02:45:01.781283 17257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53314f3399c142f1af6a3ab3558ac15a" candidate_uuid: "57c34184a77044d9ae5d7b754069936b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b220186e43e34781af1b15d2d60be62c" is_pre_election: true
I20260621 02:45:01.781476 16972 raft_consensus.cc:2454] T 53314f3399c142f1af6a3ab3558ac15a P df8a57d499a54ac5bc5d7a5579e910cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57c34184a77044d9ae5d7b754069936b in term 0.
I20260621 02:45:01.781618 17257 raft_consensus.cc:2454] T 53314f3399c142f1af6a3ab3558ac15a P b220186e43e34781af1b15d2d60be62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57c34184a77044d9ae5d7b754069936b in term 0.
I20260621 02:45:01.781960 17050 leader_election.cc:302] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57c34184a77044d9ae5d7b754069936b, df8a57d499a54ac5bc5d7a5579e910cf; no voters:
I20260621 02:45:01.782198 17399 raft_consensus.cc:2781] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:45:01.782295 17399 raft_consensus.cc:482] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:45:01.782357 17399 raft_consensus.cc:3037] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:01.784433 17399 raft_consensus.cc:504] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c34184a77044d9ae5d7b754069936b" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39959 } } peers { permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 } } peers { permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 } }
I20260621 02:45:01.784992 17399 leader_election.cc:288] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [CANDIDATE]: Term 1 election: Requested vote from peers df8a57d499a54ac5bc5d7a5579e910cf (127.9.166.1:35761), b220186e43e34781af1b15d2d60be62c (127.9.166.3:46685)
I20260621 02:45:01.785473 17257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53314f3399c142f1af6a3ab3558ac15a" candidate_uuid: "57c34184a77044d9ae5d7b754069936b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b220186e43e34781af1b15d2d60be62c"
I20260621 02:45:01.785473 16972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53314f3399c142f1af6a3ab3558ac15a" candidate_uuid: "57c34184a77044d9ae5d7b754069936b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8a57d499a54ac5bc5d7a5579e910cf"
I20260621 02:45:01.785672 17257 raft_consensus.cc:3037] T 53314f3399c142f1af6a3ab3558ac15a P b220186e43e34781af1b15d2d60be62c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:01.785672 16972 raft_consensus.cc:3037] T 53314f3399c142f1af6a3ab3558ac15a P df8a57d499a54ac5bc5d7a5579e910cf [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:01.788618 16972 raft_consensus.cc:2454] T 53314f3399c142f1af6a3ab3558ac15a P df8a57d499a54ac5bc5d7a5579e910cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57c34184a77044d9ae5d7b754069936b in term 1.
I20260621 02:45:01.788617 17257 raft_consensus.cc:2454] T 53314f3399c142f1af6a3ab3558ac15a P b220186e43e34781af1b15d2d60be62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57c34184a77044d9ae5d7b754069936b in term 1.
I20260621 02:45:01.789000 17050 leader_election.cc:302] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57c34184a77044d9ae5d7b754069936b, df8a57d499a54ac5bc5d7a5579e910cf; no voters:
I20260621 02:45:01.789305 17399 raft_consensus.cc:2781] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:45:01.789737 17399 raft_consensus.cc:686] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [term 1 LEADER]: Becoming Leader. State: Replica: 57c34184a77044d9ae5d7b754069936b, State: Running, Role: LEADER
I20260621 02:45:01.790052 17399 consensus_queue.cc:237] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c34184a77044d9ae5d7b754069936b" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39959 } } peers { permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 } } peers { permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 } }
I20260621 02:45:01.790824 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.790350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.2:60915)
0621 02:45:01.790547 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.790622 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260621 02:45:01.793368 16836 catalog_manager.cc:5495] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b reported cstate change: term changed from 0 to 1, leader changed from <none> to 57c34184a77044d9ae5d7b754069936b (127.9.166.2). New cstate: current_term: 1 leader_uuid: "57c34184a77044d9ae5d7b754069936b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c34184a77044d9ae5d7b754069936b" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 } health_report { overall_health: UNKNOWN } } }
I20260621 02:45:01.828465 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.823927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.9.166.2:39959)
0621 02:45:01.824084 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:01.824089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:01.824106 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:45:01.824227 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:01.824230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:01.824293 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:01.824373 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:45:01.824704 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.825428 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.826739 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.827197 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.827291 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:01.827767 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:45:01.827888 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:45:01.827965 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:01.828089 (+ 124us) server_negotiation.cc:299] Negotiation successful
0621 02:45:01.828150 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 02:45:01.831570 16972 raft_consensus.cc:1261] T 53314f3399c142f1af6a3ab3558ac15a P df8a57d499a54ac5bc5d7a5579e910cf [term 1 FOLLOWER]: Refusing update from remote peer 57c34184a77044d9ae5d7b754069936b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:45:01.831606 17257 raft_consensus.cc:1261] T 53314f3399c142f1af6a3ab3558ac15a P b220186e43e34781af1b15d2d60be62c [term 1 FOLLOWER]: Refusing update from remote peer 57c34184a77044d9ae5d7b754069936b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:45:01.832228 17408 consensus_queue.cc:1035] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8a57d499a54ac5bc5d7a5579e910cf" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:45:01.832547 17399 consensus_queue.cc:1035] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b220186e43e34781af1b15d2d60be62c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 46685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:45:01.835474 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.834769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.3:35939)
0621 02:45:01.835046 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.835211 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260621 02:45:01.839556 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.838994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.1:37441)
0621 02:45:01.839156 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.839296 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260621 02:45:01.839607 17412 mvcc.cc:205] Tried to move back new op lower bound from 7299112557895024640 to 7299112557733715968. Current Snapshot: MvccSnapshot[applied={T|T < 7299112557895024640}]
I20260621 02:45:01.841223 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.840769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.3:34621)
0621 02:45:01.840981 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.841067 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260621 02:45:01.842027 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.841358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.2:59785)
0621 02:45:01.841783 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.841867 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319}
I20260621 02:45:01.847929 16836 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:45:01.848184 16836 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:45:01.854703 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.853872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.2:33933)
0621 02:45:01.854438 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.854541 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449}
I20260621 02:45:01.856405 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.855881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.9.166.1:39539)
0621 02:45:01.856129 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:01.856227 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260621 02:45:01.856654 16836 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ea8bd55103d4ab7ba775a30111cd5a0: Sending DeleteTablet for 3 replicas of tablet 53314f3399c142f1af6a3ab3558ac15a
I20260621 02:45:01.857600 16952 tablet_service.cc:1514] Processing DeleteTablet for tablet 53314f3399c142f1af6a3ab3558ac15a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:01 UTC) from {username='oryx', principal='oryx/127.9.166.60@KRBTEST.COM'} at 127.0.0.1:45596
I20260621 02:45:01.858031 17097 tablet_service.cc:1514] Processing DeleteTablet for tablet 53314f3399c142f1af6a3ab3558ac15a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:01 UTC) from {username='oryx', principal='oryx/127.9.166.60@KRBTEST.COM'} at 127.0.0.1:36526
I20260621 02:45:01.858657 17238 tablet_service.cc:1514] Processing DeleteTablet for tablet 53314f3399c142f1af6a3ab3558ac15a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:01 UTC) from {username='oryx', principal='oryx/127.9.166.60@KRBTEST.COM'} at 127.0.0.1:41876
I20260621 02:45:01.859288 17423 tablet_replica.cc:330] T 53314f3399c142f1af6a3ab3558ac15a P b220186e43e34781af1b15d2d60be62c: stopping tablet replica
I20260621 02:45:01.859272 16951 tablet_service.cc:1514] Processing DeleteTablet for tablet 53314f3399c142f1af6a3ab3558ac15a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:01 UTC) from {username='oryx', principal='oryx/127.9.166.60@KRBTEST.COM'} at 127.0.0.1:45596
I20260621 02:45:01.859848 17423 raft_consensus.cc:2229] T 53314f3399c142f1af6a3ab3558ac15a P b220186e43e34781af1b15d2d60be62c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:45:01.860252 17423 raft_consensus.cc:2258] T 53314f3399c142f1af6a3ab3558ac15a P b220186e43e34781af1b15d2d60be62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:01.861215 17237 tablet_service.cc:1514] Processing DeleteTablet for tablet 53314f3399c142f1af6a3ab3558ac15a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:01 UTC) from {username='oryx', principal='oryx/127.9.166.60@KRBTEST.COM'} at 127.0.0.1:41876
I20260621 02:45:01.861538 17423 ts_tablet_manager.cc:1905] T 53314f3399c142f1af6a3ab3558ac15a P b220186e43e34781af1b15d2d60be62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:45:01.862043 17422 tablet_replica.cc:330] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b: stopping tablet replica
I20260621 02:45:01.862231 17422 raft_consensus.cc:2229] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:45:01.862607 17422 raft_consensus.cc:2258] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:01.863521 17422 ts_tablet_manager.cc:1905] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:45:01.864243 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 16826
I20260621 02:45:01.864465 17421 tablet_replica.cc:330] T 53314f3399c142f1af6a3ab3558ac15a P df8a57d499a54ac5bc5d7a5579e910cf: stopping tablet replica
I20260621 02:45:01.864630 17421 raft_consensus.cc:2229] T 53314f3399c142f1af6a3ab3558ac15a P df8a57d499a54ac5bc5d7a5579e910cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:45:01.867678 17421 raft_consensus.cc:2258] T 53314f3399c142f1af6a3ab3558ac15a P df8a57d499a54ac5bc5d7a5579e910cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:01.869045 17423 ts_tablet_manager.cc:1918] T 53314f3399c142f1af6a3ab3558ac15a P b220186e43e34781af1b15d2d60be62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:45:01.869290 17423 log.cc:1196] T 53314f3399c142f1af6a3ab3558ac15a P b220186e43e34781af1b15d2d60be62c: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/53314f3399c142f1af6a3ab3558ac15a
I20260621 02:45:01.869696 17421 ts_tablet_manager.cc:1905] T 53314f3399c142f1af6a3ab3558ac15a P df8a57d499a54ac5bc5d7a5579e910cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:45:01.869880 17423 ts_tablet_manager.cc:1939] T 53314f3399c142f1af6a3ab3558ac15a P b220186e43e34781af1b15d2d60be62c: Deleting consensus metadata
I20260621 02:45:01.870992 16821 catalog_manager.cc:4841] TS b220186e43e34781af1b15d2d60be62c (127.9.166.3:46685): tablet 53314f3399c142f1af6a3ab3558ac15a (table test-table [id=476e76d0f06e4ad6b7f46c545f552cea]) successfully deleted
I20260621 02:45:01.871752 17422 ts_tablet_manager.cc:1918] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:45:01.871879 17422 log.cc:1196] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/53314f3399c142f1af6a3ab3558ac15a
W20260621 02:45:01.871973 16821 catalog_manager.cc:4808] TS b220186e43e34781af1b15d2d60be62c (127.9.166.3:46685): delete failed for tablet 53314f3399c142f1af6a3ab3558ac15a because the tablet was not found. No further retry: Not found: Tablet not found: 53314f3399c142f1af6a3ab3558ac15a
I20260621 02:45:01.872292 17422 ts_tablet_manager.cc:1939] T 53314f3399c142f1af6a3ab3558ac15a P 57c34184a77044d9ae5d7b754069936b: Deleting consensus metadata
I20260621 02:45:01.873963 16820 catalog_manager.cc:4841] TS 57c34184a77044d9ae5d7b754069936b (127.9.166.2:39959): tablet 53314f3399c142f1af6a3ab3558ac15a (table test-table [id=476e76d0f06e4ad6b7f46c545f552cea]) successfully deleted
I20260621 02:45:01.876293 17421 ts_tablet_manager.cc:1918] T 53314f3399c142f1af6a3ab3558ac15a P df8a57d499a54ac5bc5d7a5579e910cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:45:01.876423 17421 log.cc:1196] T 53314f3399c142f1af6a3ab3558ac15a P df8a57d499a54ac5bc5d7a5579e910cf: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/53314f3399c142f1af6a3ab3558ac15a
I20260621 02:45:01.876854 17421 ts_tablet_manager.cc:1939] T 53314f3399c142f1af6a3ab3558ac15a P df8a57d499a54ac5bc5d7a5579e910cf: Deleting consensus metadata
I20260621 02:45:01.877799 16820 catalog_manager.cc:4841] TS df8a57d499a54ac5bc5d7a5579e910cf (127.9.166.1:35761): tablet 53314f3399c142f1af6a3ab3558ac15a (table test-table [id=476e76d0f06e4ad6b7f46c545f552cea]) successfully deleted
W20260621 02:45:01.878172 16820 catalog_manager.cc:4808] TS df8a57d499a54ac5bc5d7a5579e910cf (127.9.166.1:35761): delete failed for tablet 53314f3399c142f1af6a3ab3558ac15a because the tablet was not found. No further retry: Not found: Tablet not found: 53314f3399c142f1af6a3ab3558ac15a
I20260621 02:45:01.882537 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:01.864955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45598 (local address 127.9.166.1:35761)
0621 02:45:01.865121 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:01.865126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:01.867565 (+ 2439us) server_negotiation.cc:407] Connection header received
0621 02:45:01.867833 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:01.867838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:01.867913 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:01.868010 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:01.869931 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.871046 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.872832 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:01.873511 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:01.873601 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:01.877170 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:01.877183 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:01.877204 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:01.877245 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:01.880288 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:01.880916 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:01.880924 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:01.880928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:01.881045 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:01.881335 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:01.881338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:01.881340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:01.881706 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:01.881917 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:01.882123 (+ 206us) server_negotiation.cc:299] Negotiation successful
0621 02:45:01.882274 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 02:45:01.883261 17012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:01.986232 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 16826
I20260621 02:45:02.003263 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 17029
I20260621 02:45:02.016314 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:02.004037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36544 (local address 127.9.166.2:39959)
0621 02:45:02.004226 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:02.004231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:02.004249 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:45:02.004384 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:02.004388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:02.004453 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:02.004542 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:02.005730 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:02.006465 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:02.007804 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:02.008240 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:02.008338 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:02.010675 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:02.010690 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:02.010705 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:02.010735 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:02.013914 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:02.014581 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:02.014588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:02.014590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:02.014707 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:02.015014 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:02.015017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:02.015019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:02.015329 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:02.015634 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:02.015802 (+ 168us) server_negotiation.cc:299] Negotiation successful
0621 02:45:02.015948 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 02:45:02.016928 17157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:02.134572 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 17029
I20260621 02:45:02.156970 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 17175
I20260621 02:45:02.169507 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:02.157680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41882 (local address 127.9.166.3:46685)
0621 02:45:02.157895 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:02.157900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:02.157917 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:45:02.158023 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:02.158027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:02.158117 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:02.158201 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:02.159484 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:02.160296 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:02.161536 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:02.161932 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:02.162023 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:02.164522 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:02.164537 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:02.164553 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:02.164601 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:02.167207 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:02.167808 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:02.167815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:02.167817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:02.167935 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:02.168241 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:02.168244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:02.168245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:02.168573 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:02.168768 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:02.169020 (+ 252us) server_negotiation.cc:299] Negotiation successful
0621 02:45:02.169177 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:45:02.170177 17298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:02.255851 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:02.255078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.0.0.1:42916)
0621 02:45:02.255584 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:02.255678 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":272,"threads_started":1}
I20260621 02:45:02.259528 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:02.259139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37723 (local address 127.0.0.1:42922)
0621 02:45:02.259283 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:02.259361 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:37723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260621 02:45:02.264767 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 17175
I20260621 02:45:02.281875 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 16497
I20260621 02:45:02.282927 16669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:02.399935 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 16497
I20260621 02:45:02.420686 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 16595
I20260621 02:45:02.421778 16867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:02.543944 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 16595
2026-06-21T02:45:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9431 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:45:02 dist-test-slave-hvgn krb5kdc[17434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:45:02 dist-test-slave-hvgn krb5kdc[17434](info): set up 2 sockets
Jun 21 02:45:02 dist-test-slave-hvgn krb5kdc[17434](info): commencing operation
krb5kdc: starting...
W20260621 02:45:03.636292 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:45:03 dist-test-slave-hvgn krb5kdc[17434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:45:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:45:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.62.keytab.
Entry for principal oryx/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.62.keytab.
I20260621 02:45:03.842602 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:38891
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.62.keytab
--principal=oryx/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:45639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:38891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:45:04.164053 17447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:04.164342 17447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:04.164491 17447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:04.171671 17447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:45:04.171757 17447 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260621 02:45:04.171844 17447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:04.171954 17447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:04.172036 17447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:45:04.172144 17447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:45:04.181631 17447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:45639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:38891
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.62.keytab
--principal=oryx/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:38891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:04.182863 17447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:04.184183 17447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:04.195792 17455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:04.195859 17453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:04.195895 17452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:04.196648 17447 server_base.cc:1034] running on GCE node
I20260621 02:45:04.197362 17447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:04.198709 17447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:04.199940 17447 hybrid_clock.cc:648] HybridClock initialized: now 1782009904199872 us; error 47 us; skew 500 ppm
I20260621 02:45:04.200274 17447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:45:04 dist-test-slave-hvgn krb5kdc[17434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009904, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:45:04.203145 17447 init.cc:377] Logged in from keytab as oryx/127.9.166.62@KRBTEST.COM (short username oryx)
I20260621 02:45:04.204622 17447 webserver.cc:460] Webserver started at http://127.9.166.62:34427/ using document root <none> and password file <none>
I20260621 02:45:04.205292 17447 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:04.205420 17447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:04.205734 17447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:45:04.208559 17447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "e7bb31d65aae4bfd8d9ff296a1a9cda5"
format_stamp: "Formatted at 2026-06-21 02:45:04 on dist-test-slave-hvgn"
server_key: "fe08120828016453a3dff7515cfb5320"
server_key_iv: "29b11f72bb84d56acdc1c55e32e5740b"
server_key_version: "encryptionkey@0"
I20260621 02:45:04.209326 17447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "e7bb31d65aae4bfd8d9ff296a1a9cda5"
format_stamp: "Formatted at 2026-06-21 02:45:04 on dist-test-slave-hvgn"
server_key: "fe08120828016453a3dff7515cfb5320"
server_key_iv: "29b11f72bb84d56acdc1c55e32e5740b"
server_key_version: "encryptionkey@0"
I20260621 02:45:04.214066 17447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260621 02:45:04.217410 17462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:04.218688 17447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 02:45:04.218855 17447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "e7bb31d65aae4bfd8d9ff296a1a9cda5"
format_stamp: "Formatted at 2026-06-21 02:45:04 on dist-test-slave-hvgn"
server_key: "fe08120828016453a3dff7515cfb5320"
server_key_iv: "29b11f72bb84d56acdc1c55e32e5740b"
server_key_version: "encryptionkey@0"
I20260621 02:45:04.219064 17447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:04.235262 17447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:04.239042 17447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:04.239310 17447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:04.260716 17447 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:38891
I20260621 02:45:04.260735 17523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:38891 every 8 connection(s)
I20260621 02:45:04.262069 17447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:45:04.265805 17524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:04.271855 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 17447
I20260621 02:45:04.272061 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:45:04.272666 9880 external_mini_cluster.cc:1428] Setting key d4223822022b4e7989f5dd7b76d1790a
I20260621 02:45:04.276373 17524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5: Bootstrap starting.
I20260621 02:45:04.279903 17524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:04.280915 17524 log.cc:824] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5: Log is configured to *not* fsync() on all Append() calls
I20260621 02:45:04.283846 17524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5: No bootstrap required, opened a new log
Jun 21 02:45:04 dist-test-slave-hvgn krb5kdc[17434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM
I20260621 02:45:04.292697 17524 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7bb31d65aae4bfd8d9ff296a1a9cda5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38891 } }
I20260621 02:45:04.293109 17524 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:04.293219 17524 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7bb31d65aae4bfd8d9ff296a1a9cda5, State: Initialized, Role: FOLLOWER
I20260621 02:45:04.293895 17524 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7bb31d65aae4bfd8d9ff296a1a9cda5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38891 } }
I20260621 02:45:04.294173 17524 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:45:04.294312 17524 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:45:04.294476 17524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:04.295415 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:04.274608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40960 (local address 127.9.166.62:38891)
0621 02:45:04.275473 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:04.275485 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:04.275525 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:45:04.277220 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:04.277241 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:04.277605 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:04.277908 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:04.279669 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:04.281044 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:04.282593 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:04.283230 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:04.283361 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:04.287902 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:04.287912 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:04.287926 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:04.287967 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:04.291060 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:04.291802 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:04.291815 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:04.291822 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:04.291995 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:04.292369 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:04.292374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:04.292376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:04.292859 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:04.293156 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:04.293567 (+ 411us) server_negotiation.cc:299] Negotiation successful
0621 02:45:04.293997 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":360,"threads_started":1}
I20260621 02:45:04.297088 17524 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7bb31d65aae4bfd8d9ff296a1a9cda5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38891 } }
I20260621 02:45:04.297564 17524 leader_election.cc:302] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7bb31d65aae4bfd8d9ff296a1a9cda5; no voters:
I20260621 02:45:04.298187 17524 leader_election.cc:288] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:45:04.298300 17529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:45:04.298997 17529 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5 [term 1 LEADER]: Becoming Leader. State: Replica: e7bb31d65aae4bfd8d9ff296a1a9cda5, State: Running, Role: LEADER
I20260621 02:45:04.299587 17529 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7bb31d65aae4bfd8d9ff296a1a9cda5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38891 } }
I20260621 02:45:04.300483 17524 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:45:04.302299 17530 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7bb31d65aae4bfd8d9ff296a1a9cda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7bb31d65aae4bfd8d9ff296a1a9cda5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38891 } } }
I20260621 02:45:04.302687 17530 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5 [sys.catalog]: This master's current role is: LEADER
I20260621 02:45:04.303472 17531 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7bb31d65aae4bfd8d9ff296a1a9cda5. Latest consensus state: current_term: 1 leader_uuid: "e7bb31d65aae4bfd8d9ff296a1a9cda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7bb31d65aae4bfd8d9ff296a1a9cda5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38891 } } }
I20260621 02:45:04.303684 17534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:45:04.303810 17531 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5 [sys.catalog]: This master's current role is: LEADER
I20260621 02:45:04.310261 17534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:45:04.319754 17534 catalog_manager.cc:1347] Generated new cluster ID: 05c63e849f3a4c509cfd48fa96f104c5
I20260621 02:45:04.319861 17534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:45:04.338394 17534 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:45:04.339802 17534 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:45:04.351487 17534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5: Generated new TSK 0
I20260621 02:45:04.352480 17534 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.1.keytab.
Entry for principal oryx/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.1.keytab.
I20260621 02:45:04.408277 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:38891
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.1.keytab
--principal=oryx/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:45639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:45:04.731070 17550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:04.731383 17550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:04.731580 17550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:04.740065 17550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:04.740204 17550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:04.740360 17550 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:45:04.740494 17550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:45:04.750423 17550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:45639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.1.keytab
--principal=oryx/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.166.62:38891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:04.751720 17550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:04.753006 17550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:04.765829 17555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:04.765841 17556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:04.765935 17558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:04.766821 17550 server_base.cc:1034] running on GCE node
I20260621 02:45:04.767520 17550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:04.768462 17550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:04.769699 17550 hybrid_clock.cc:648] HybridClock initialized: now 1782009904769636 us; error 33 us; skew 500 ppm
I20260621 02:45:04.770090 17550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:45:04 dist-test-slave-hvgn krb5kdc[17434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009904, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:45:04.773524 17550 init.cc:377] Logged in from keytab as oryx/127.9.166.1@KRBTEST.COM (short username oryx)
I20260621 02:45:04.774924 17550 webserver.cc:460] Webserver started at http://127.9.166.1:41341/ using document root <none> and password file <none>
I20260621 02:45:04.775702 17550 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:04.775807 17550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:04.776085 17550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:45:04.779052 17550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "8460ee014fce4c72935c507fb0a96606"
format_stamp: "Formatted at 2026-06-21 02:45:04 on dist-test-slave-hvgn"
server_key: "ae4feb73026efdc18259f3f8c823d402"
server_key_iv: "83604e3dffe98779f2aa0e8167b296f6"
server_key_version: "encryptionkey@0"
I20260621 02:45:04.779894 17550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "8460ee014fce4c72935c507fb0a96606"
format_stamp: "Formatted at 2026-06-21 02:45:04 on dist-test-slave-hvgn"
server_key: "ae4feb73026efdc18259f3f8c823d402"
server_key_iv: "83604e3dffe98779f2aa0e8167b296f6"
server_key_version: "encryptionkey@0"
I20260621 02:45:04.785018 17550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:45:04.788384 17565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:04.789803 17550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:45:04.790069 17550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "8460ee014fce4c72935c507fb0a96606"
format_stamp: "Formatted at 2026-06-21 02:45:04 on dist-test-slave-hvgn"
server_key: "ae4feb73026efdc18259f3f8c823d402"
server_key_iv: "83604e3dffe98779f2aa0e8167b296f6"
server_key_version: "encryptionkey@0"
I20260621 02:45:04.790292 17550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:04.814798 17550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:04.818823 17550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:04.819092 17550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:04.821681 17550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:45:04.821795 17550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:04.821962 17550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:45:04.822050 17550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:45:04 dist-test-slave-hvgn krb5kdc[17434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009904, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM
I20260621 02:45:04.845702 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:04.822450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:54373 (local address 127.9.166.62:38891)
0621 02:45:04.823015 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:04.823021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:04.824193 (+ 1172us) server_negotiation.cc:407] Connection header received
0621 02:45:04.826176 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:04.826185 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:04.826295 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:04.826446 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:04.828321 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:04.829413 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:04.832037 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:04.832575 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:04.832722 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:04.838081 (+ 5359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:04.838088 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:04.838093 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:04.838142 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:04.842054 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:04.842772 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:04.842781 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:04.842787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:04.842981 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:04.843428 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:04.843436 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:04.843442 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:04.843855 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:04.844094 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:04.845101 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0621 02:45:04.845322 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":324,"threads_started":1}
I20260621 02:45:04.847231 17575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:04.823105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38891 (local address 127.9.166.1:54373)
0621 02:45:04.824028 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:04.824076 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:04.825894 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:04.826611 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:04.826618 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:04.827280 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:04.828141 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:04.828163 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:04.829638 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:04.829648 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:04.831779 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:04.831790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:04.832820 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:04.832828 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:04.833175 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:04.834188 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:04.834230 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:04.837805 (+ 3575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:45:04.842210 (+ 4405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:04.842223 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:04.842236 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:04.842638 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:04.843098 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:04.843101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:04.843103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:04.843293 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:04.844067 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:45:04.844082 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:45:04.844531 (+ 449us) client_negotiation.cc:769] Sending connection context
0621 02:45:04.844969 (+ 438us) client_negotiation.cc:240] Negotiation successful
0621 02:45:04.845531 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":376,"threads_started":1}
I20260621 02:45:04.875092 17550 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:32967
I20260621 02:45:04.875178 17685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:32967 every 8 connection(s)
I20260621 02:45:04.876014 17642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:04.858681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38891 (local address 127.9.166.1:48689)
0621 02:45:04.859845 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:04.859867 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:04.860048 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:04.860450 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:04.860455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:04.860871 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:04.861142 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:04.861151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:04.862739 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:04.862746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:04.865033 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:04.865052 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:04.866248 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:04.866259 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:04.866394 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:04.867064 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:04.867110 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:04.868510 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:45:04.872892 (+ 4382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:04.872898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:04.872905 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:04.873419 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:04.873880 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:04.873885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:04.873890 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:04.874053 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:04.874765 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:45:04.874770 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:45:04.875032 (+ 262us) client_negotiation.cc:769] Sending connection context
0621 02:45:04.875194 (+ 162us) client_negotiation.cc:240] Negotiation successful
0621 02:45:04.875558 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"thread_start_us":813,"threads_started":1}
I20260621 02:45:04.876323 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:04.858589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:48689 (local address 127.9.166.62:38891)
0621 02:45:04.858790 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:04.858798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:04.859989 (+ 1191us) server_negotiation.cc:407] Connection header received
0621 02:45:04.860190 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:04.860199 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:04.860290 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:04.860401 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:04.861280 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:04.862549 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:04.865208 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:04.866015 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:04.866157 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:04.868743 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:04.868750 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:04.868757 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:04.868802 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:04.872741 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:04.873552 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:04.873558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:04.873561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:04.873731 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:04.874199 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:04.874207 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:04.874213 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:04.874631 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:04.874786 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:04.875620 (+ 834us) server_negotiation.cc:299] Negotiation successful
0621 02:45:04.875843 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:45:04.876593 17550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:45:04.879659 17686 heartbeater.cc:347] Connected to a master server at 127.9.166.62:38891
I20260621 02:45:04.879911 17686 heartbeater.cc:464] Registering TS with master...
I20260621 02:45:04.880146 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 17550
I20260621 02:45:04.880339 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:45:04.880651 17686 heartbeater.cc:511] Master 127.9.166.62:38891 requested a full tablet report, sending...
I20260621 02:45:04.881024 9880 external_mini_cluster.cc:1428] Setting key 8465c1592844d7eba873d9d2e209fe28
I20260621 02:45:04.882745 17479 ts_manager.cc:194] Registered new tserver with Master: 8460ee014fce4c72935c507fb0a96606 (127.9.166.1:32967)
I20260621 02:45:04.884250 17479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:54373
WARNING: no policy specified for oryx/127.9.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.2.keytab.
Entry for principal oryx/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.2.keytab.
I20260621 02:45:04.928018 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:38891
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.2.keytab
--principal=oryx/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:45639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:45:05.239735 17690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:05.240046 17690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:05.240216 17690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:05.247231 17690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:05.247370 17690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:05.247536 17690 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:45:05.247669 17690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:45:05.257125 17690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:45639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.2.keytab
--principal=oryx/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.166.62:38891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:05.258342 17690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:05.259717 17690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:05.276312 17695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:05.278024 17696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:05.280222 17698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:05.280447 17690 server_base.cc:1034] running on GCE node
I20260621 02:45:05.281194 17690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:05.282300 17690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:05.283540 17690 hybrid_clock.cc:648] HybridClock initialized: now 1782009905283453 us; error 63 us; skew 500 ppm
I20260621 02:45:05.283922 17690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:45:05 dist-test-slave-hvgn krb5kdc[17434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009905, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:45:05.287580 17690 init.cc:377] Logged in from keytab as oryx/127.9.166.2@KRBTEST.COM (short username oryx)
I20260621 02:45:05.289007 17690 webserver.cc:460] Webserver started at http://127.9.166.2:33787/ using document root <none> and password file <none>
I20260621 02:45:05.289750 17690 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:05.289881 17690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:05.290251 17690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:45:05.293213 17690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "cf48897dfda844d685b5b2540558361c"
format_stamp: "Formatted at 2026-06-21 02:45:05 on dist-test-slave-hvgn"
server_key: "1c778df77d53c1dc6ef3393f09095db1"
server_key_iv: "96d6872ae2b1bb40be34aabcf9726024"
server_key_version: "encryptionkey@0"
I20260621 02:45:05.294046 17690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "cf48897dfda844d685b5b2540558361c"
format_stamp: "Formatted at 2026-06-21 02:45:05 on dist-test-slave-hvgn"
server_key: "1c778df77d53c1dc6ef3393f09095db1"
server_key_iv: "96d6872ae2b1bb40be34aabcf9726024"
server_key_version: "encryptionkey@0"
I20260621 02:45:05.299162 17690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260621 02:45:05.302683 17705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:05.304287 17690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:45:05.304447 17690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "cf48897dfda844d685b5b2540558361c"
format_stamp: "Formatted at 2026-06-21 02:45:05 on dist-test-slave-hvgn"
server_key: "1c778df77d53c1dc6ef3393f09095db1"
server_key_iv: "96d6872ae2b1bb40be34aabcf9726024"
server_key_version: "encryptionkey@0"
I20260621 02:45:05.304697 17690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:05.323797 17690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:05.327982 17690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:05.328258 17690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:05.330766 17690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:45:05.330874 17690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:05.331044 17690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:45:05.331142 17690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:45:05 dist-test-slave-hvgn krb5kdc[17434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009905, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM
I20260621 02:45:05.357023 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:05.331507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:52323 (local address 127.9.166.62:38891)
0621 02:45:05.331709 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:05.331715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:05.333156 (+ 1441us) server_negotiation.cc:407] Connection header received
0621 02:45:05.335675 (+ 2519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:05.335683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:05.335809 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:05.335894 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:05.338359 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:05.339452 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:05.342763 (+ 3311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:05.343320 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:05.343521 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:05.348855 (+ 5334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:05.348865 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:05.348872 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:05.348924 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:05.352999 (+ 4075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:05.354026 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:05.354034 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:05.354040 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:05.354191 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:05.354870 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:05.354875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:05.354878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:05.355261 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:05.355432 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:05.356440 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0621 02:45:05.356678 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:45:05.358922 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:05.332080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38891 (local address 127.9.166.2:52323)
0621 02:45:05.332960 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:05.333012 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:05.335259 (+ 2247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:05.336495 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:05.336506 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:05.337270 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:05.338181 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:05.338215 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:05.339617 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:05.339627 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:05.342577 (+ 2950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:05.342596 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:05.343667 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:05.343675 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:05.344122 (+ 447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:05.344979 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:05.345038 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:05.348602 (+ 3564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:45:05.353175 (+ 4573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:05.353182 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:05.353196 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:05.353877 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:05.354319 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:05.354326 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:05.354329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:05.354702 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:05.355483 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:45:05.355491 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:45:05.356008 (+ 517us) client_negotiation.cc:769] Sending connection context
0621 02:45:05.356362 (+ 354us) client_negotiation.cc:240] Negotiation successful
0621 02:45:05.356934 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":389,"threads_started":1}
I20260621 02:45:05.373751 17690 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:37627
I20260621 02:45:05.373806 17824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:37627 every 8 connection(s)
I20260621 02:45:05.375126 17690 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:45:05.377789 17825 heartbeater.cc:347] Connected to a master server at 127.9.166.62:38891
I20260621 02:45:05.378077 17825 heartbeater.cc:464] Registering TS with master...
I20260621 02:45:05.378942 17825 heartbeater.cc:511] Master 127.9.166.62:38891 requested a full tablet report, sending...
I20260621 02:45:05.380446 17479 ts_manager.cc:194] Registered new tserver with Master: cf48897dfda844d685b5b2540558361c (127.9.166.2:37627)
I20260621 02:45:05.380522 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 17690
I20260621 02:45:05.380671 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:45:05.381039 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:05.365641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:59863 (local address 127.9.166.62:38891)
0621 02:45:05.365837 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:05.365847 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:05.366819 (+ 972us) server_negotiation.cc:407] Connection header received
0621 02:45:05.367018 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:05.367027 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:05.367144 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:05.367275 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:05.368783 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:05.369909 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:05.372165 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:05.372769 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:05.372883 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:05.374880 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:05.374887 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:05.374891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:05.374926 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:05.377991 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:05.378791 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:05.378795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:05.378797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:05.378913 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:05.379290 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:05.379293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:05.379295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:05.379644 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:05.379772 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:05.380469 (+ 697us) server_negotiation.cc:299] Negotiation successful
0621 02:45:05.380696 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:45:05.381181 17804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:05.365866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38891 (local address 127.9.166.2:59863)
0621 02:45:05.366686 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:05.366712 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:05.366875 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:05.367291 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:05.367294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:05.367984 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:05.368557 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:05.368576 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:05.370092 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:05.370095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:05.372030 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:05.372042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:05.373107 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:05.373112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:05.373222 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:05.373901 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:05.373925 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:05.374684 (+ 759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:45:05.378106 (+ 3422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:05.378113 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:05.378119 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:05.378667 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:05.379025 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:05.379029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:05.379033 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:05.379180 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:05.379914 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:45:05.379919 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:45:05.380187 (+ 268us) client_negotiation.cc:769] Sending connection context
0621 02:45:05.380372 (+ 185us) client_negotiation.cc:240] Negotiation successful
0621 02:45:05.380702 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":475,"threads_started":1}
I20260621 02:45:05.381206 9880 external_mini_cluster.cc:1428] Setting key 365da7dd5779ebf644d913152323779b
I20260621 02:45:05.381580 17479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:52323
WARNING: no policy specified for oryx/127.9.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.3.keytab.
Entry for principal oryx/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.3.keytab.
I20260621 02:45:05.424576 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:38891
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.3.keytab
--principal=oryx/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:45639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:45:05.745584 17829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:05.745850 17829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:05.745946 17829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:05.753073 17829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:05.753271 17829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:05.753418 17829 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260621 02:45:05.753593 17829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:45:05.764333 17829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:45639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.3.keytab
--principal=oryx/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.166.62:38891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:05.765566 17829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:05.766796 17829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:05.779917 17834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:05.780196 17835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:05.780826 17837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:05.781208 17829 server_base.cc:1034] running on GCE node
I20260621 02:45:05.781889 17829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:05.782835 17829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:05.784060 17829 hybrid_clock.cc:648] HybridClock initialized: now 1782009905783987 us; error 38 us; skew 500 ppm
I20260621 02:45:05.784405 17829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:45:05 dist-test-slave-hvgn krb5kdc[17434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009905, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:45:05.787626 17829 init.cc:377] Logged in from keytab as oryx/127.9.166.3@KRBTEST.COM (short username oryx)
I20260621 02:45:05.789109 17829 webserver.cc:460] Webserver started at http://127.9.166.3:33217/ using document root <none> and password file <none>
I20260621 02:45:05.789849 17829 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:05.789986 17829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:05.790343 17829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:45:05.793440 17829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "b20a1a36b827478fb586a00ec935b92b"
format_stamp: "Formatted at 2026-06-21 02:45:05 on dist-test-slave-hvgn"
server_key: "de56af013abffc6363d266df9b1b1ba9"
server_key_iv: "880dca63b83e6a87aee066ae48aea78f"
server_key_version: "encryptionkey@0"
I20260621 02:45:05.794305 17829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "b20a1a36b827478fb586a00ec935b92b"
format_stamp: "Formatted at 2026-06-21 02:45:05 on dist-test-slave-hvgn"
server_key: "de56af013abffc6363d266df9b1b1ba9"
server_key_iv: "880dca63b83e6a87aee066ae48aea78f"
server_key_version: "encryptionkey@0"
I20260621 02:45:05.800118 17829 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 02:45:05.803808 17844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:05.805238 17829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260621 02:45:05.805442 17829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "b20a1a36b827478fb586a00ec935b92b"
format_stamp: "Formatted at 2026-06-21 02:45:05 on dist-test-slave-hvgn"
server_key: "de56af013abffc6363d266df9b1b1ba9"
server_key_iv: "880dca63b83e6a87aee066ae48aea78f"
server_key_version: "encryptionkey@0"
I20260621 02:45:05.805686 17829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:05.845427 17829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:05.850018 17829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:05.850332 17829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:05.853612 17829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:45:05.853724 17829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:05.853879 17829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:45:05.853961 17829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:45:05 dist-test-slave-hvgn krb5kdc[17434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009905, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM
I20260621 02:45:05.880435 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:05.855207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38891 (local address 127.9.166.3:48975)
0621 02:45:05.856112 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:05.856168 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:05.858393 (+ 2225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:05.859590 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:05.859598 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:05.860532 (+ 934us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:05.861501 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:05.861526 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:05.862973 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:05.862978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:05.865375 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:05.865387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:05.866455 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:05.866467 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:05.866676 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:05.867520 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:05.867546 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:05.870322 (+ 2776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:45:05.873883 (+ 3561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:05.873892 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:05.873910 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:05.874612 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:05.875195 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:05.875207 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:05.875214 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:05.875567 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:05.876459 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:45:05.876477 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:45:05.877088 (+ 611us) client_negotiation.cc:769] Sending connection context
0621 02:45:05.877442 (+ 354us) client_negotiation.cc:240] Negotiation successful
0621 02:45:05.878193 (+ 751us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":432,"threads_started":1}
I20260621 02:45:05.882169 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:05.854688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:48975 (local address 127.9.166.62:38891)
0621 02:45:05.854922 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:05.854929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:05.856296 (+ 1367us) server_negotiation.cc:407] Connection header received
0621 02:45:05.858759 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:05.858766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:05.858875 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:05.858962 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:05.861712 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:05.862795 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:05.865520 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:05.866109 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:05.866254 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:05.870542 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:05.870547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:05.870550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:05.870582 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:05.873710 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:05.874785 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:05.874794 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:05.874801 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:05.875012 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:05.875787 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:05.875793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:05.875797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:05.876253 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:05.881009 (+ 4756us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:05.881250 (+ 241us) server_negotiation.cc:299] Negotiation successful
0621 02:45:05.881596 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:45:05.887430 17686 heartbeater.cc:503] Master 127.9.166.62:38891 was elected leader, sending a full tablet report...
I20260621 02:45:05.906220 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:05.889651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:34267 (local address 127.9.166.62:38891)
0621 02:45:05.889832 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:05.889840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:05.890250 (+ 410us) server_negotiation.cc:407] Connection header received
0621 02:45:05.890446 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:05.890451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:05.890533 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:05.890642 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:05.892531 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:05.893230 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:05.896064 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:05.896532 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:05.896675 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:05.899816 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:05.899822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:05.899825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:05.899857 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:05.903298 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:05.904048 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:05.904054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:05.904060 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:05.904190 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:05.904611 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:05.904621 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:05.904623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:05.904896 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:05.905059 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:05.905543 (+ 484us) server_negotiation.cc:299] Negotiation successful
0621 02:45:05.905844 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 02:45:05.906383 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:05.889399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38891 (local address 127.9.166.3:34267)
0621 02:45:05.890152 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:05.890170 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:05.890322 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:05.890692 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:05.890702 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:05.891658 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:05.892354 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:05.892373 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:05.893370 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:05.893380 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:05.895830 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:05.895873 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:05.896770 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:05.896781 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:05.897100 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:05.898039 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:05.898071 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:05.899579 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:45:05.903471 (+ 3892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:05.903480 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:05.903484 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:05.903914 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:05.904339 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:05.904344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:05.904349 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:05.904485 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:05.905020 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:45:05.905026 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:45:05.905230 (+ 204us) client_negotiation.cc:769] Sending connection context
0621 02:45:05.905417 (+ 187us) client_negotiation.cc:240] Negotiation successful
0621 02:45:05.905768 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":505,"threads_started":1}
I20260621 02:45:05.916556 17829 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:43225
I20260621 02:45:05.916584 17963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:43225 every 8 connection(s)
I20260621 02:45:05.917874 17829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:45:05.919927 17964 heartbeater.cc:347] Connected to a master server at 127.9.166.62:38891
I20260621 02:45:05.920117 17964 heartbeater.cc:464] Registering TS with master...
I20260621 02:45:05.920711 17964 heartbeater.cc:511] Master 127.9.166.62:38891 requested a full tablet report, sending...
I20260621 02:45:05.921988 17479 ts_manager.cc:194] Registered new tserver with Master: b20a1a36b827478fb586a00ec935b92b (127.9.166.3:43225)
I20260621 02:45:05.922986 17479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:48975
I20260621 02:45:05.925071 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 17829
I20260621 02:45:05.925253 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:45:05.925733 9880 external_mini_cluster.cc:1428] Setting key f47c852b1095d64949f84cf5b1313183
I20260621 02:45:05.928398 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 02:45:05 dist-test-slave-hvgn krb5kdc[17434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009905, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 21 02:45:05 dist-test-slave-hvgn krb5kdc[17434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:05 dist-test-slave-hvgn krb5kdc[17434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 02:45:05.978909 17574 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:05.967548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40976 (local address 127.9.166.62:38891)
0621 02:45:05.967785 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:05.967792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:05.967994 (+ 202us) server_negotiation.cc:407] Connection header received
0621 02:45:05.968234 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:05.968239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:05.968306 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:05.968419 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:05.969830 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:05.970644 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:05.972068 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:05.972556 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:05.972703 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:05.978566 (+ 5863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40976: BlockingRecv error: recv got EOF from 127.0.0.1:40976 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 21 02:45:05 dist-test-slave-hvgn krb5kdc[17434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009905, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM
I20260621 02:45:05.995994 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:05.981744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40980 (local address 127.9.166.62:38891)
0621 02:45:05.981984 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:05.981993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:05.982262 (+ 269us) server_negotiation.cc:407] Connection header received
0621 02:45:05.982500 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:05.982505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:05.982592 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:05.982701 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:05.984027 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:05.984793 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:05.986117 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:05.986565 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:05.986698 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:05.990526 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:05.990535 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:05.990545 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:05.990578 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:05.993232 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:05.993939 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:05.993947 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:05.993956 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:05.994074 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:05.994512 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:05.994518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:05.994523 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:05.994937 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:05.995150 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:05.995371 (+ 221us) server_negotiation.cc:299] Negotiation successful
0621 02:45:05.995657 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260621 02:45:06.000065 17479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:45:06.007202 17479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:45:06.043344 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.019253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:43225 (local address 127.0.0.1:51578)
0621 02:45:06.032817 (+ 13564us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:06.032839 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:06.032972 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:06.034094 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:06.034102 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:06.034137 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:45:06.034814 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:06.034830 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.037589 (+ 2759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.037598 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:06.040611 (+ 3013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:06.040625 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.042253 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.042259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:06.042448 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.042511 (+ 63us) client_negotiation.cc:769] Sending connection context
0621 02:45:06.042638 (+ 127us) client_negotiation.cc:240] Negotiation successful
0621 02:45:06.042838 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13455,"mutex_wait_us":62,"thread_start_us":13300,"threads_started":1}
I20260621 02:45:06.044122 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.019917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51578 (local address 127.9.166.3:43225)
0621 02:45:06.032135 (+ 12218us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:06.032147 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:06.033050 (+ 903us) server_negotiation.cc:407] Connection header received
0621 02:45:06.033165 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:06.033171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:06.033490 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:06.033678 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:45:06.035096 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.037385 (+ 2289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.040777 (+ 3392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.042064 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.042191 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.042420 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:06.043615 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0621 02:45:06.043797 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11996,"thread_start_us":12236,"threads_started":1}
I20260621 02:45:06.044297 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.019294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35826 (local address 127.9.166.1:32967)
0621 02:45:06.031915 (+ 12621us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:06.031930 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:06.033175 (+ 1245us) server_negotiation.cc:407] Connection header received
0621 02:45:06.033278 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:06.033288 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:06.033523 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:06.033737 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:45:06.035043 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.037096 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.041499 (+ 4403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.042674 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.042803 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.042989 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:06.043820 (+ 831us) server_negotiation.cc:299] Negotiation successful
0621 02:45:06.044017 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12386,"thread_start_us":12639,"threads_started":1}
I20260621 02:45:06.045022 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.019073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:32967 (local address 127.0.0.1:35826)
0621 02:45:06.032742 (+ 13669us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:06.032780 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:06.032973 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:06.034051 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:06.034062 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:06.034102 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:45:06.034767 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:06.034784 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.037391 (+ 2607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.037398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:06.041304 (+ 3906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:06.041318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.043141 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.043150 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:06.043347 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.043414 (+ 67us) client_negotiation.cc:769] Sending connection context
0621 02:45:06.044479 (+ 1065us) client_negotiation.cc:240] Negotiation successful
0621 02:45:06.044624 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13396,"mutex_wait_us":1034,"spinlock_wait_cycles":7040,"thread_start_us":13250,"threads_started":1}
I20260621 02:45:06.046950 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.033390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:37627 (local address 127.0.0.1:51850)
0621 02:45:06.039680 (+ 6290us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:06.039696 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:06.039801 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:06.040799 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:06.040804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:06.040828 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:45:06.041234 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:06.041244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.042950 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.042956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:06.044860 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:06.044871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.046209 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.046216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:06.046399 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.046430 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 02:45:06.046513 (+ 83us) client_negotiation.cc:240] Negotiation successful
0621 02:45:06.046626 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6126,"thread_start_us":5990,"threads_started":1}
I20260621 02:45:06.048153 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.033494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.9.166.2:37627)
0621 02:45:06.040269 (+ 6775us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:06.040277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:06.040302 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:45:06.040390 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:06.040397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:06.040624 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:06.040823 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:45:06.041417 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.042780 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.045008 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.046018 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.047462 (+ 1444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.047658 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:06.047725 (+ 67us) server_negotiation.cc:299] Negotiation successful
0621 02:45:06.047871 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6618,"thread_start_us":6503,"threads_started":1}
I20260621 02:45:06.047849 17894 tablet_service.cc:1467] Processing CreateTablet for tablet eb6000d7534042f3b00f235a1d803263 (DEFAULT_TABLE table=test-table [id=119f9c0f3d0645219de3c7bcd8e3d57f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:45:06.049140 17894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb6000d7534042f3b00f235a1d803263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:06.048856 17616 tablet_service.cc:1467] Processing CreateTablet for tablet eb6000d7534042f3b00f235a1d803263 (DEFAULT_TABLE table=test-table [id=119f9c0f3d0645219de3c7bcd8e3d57f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:45:06.050119 17616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb6000d7534042f3b00f235a1d803263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:06.051350 17755 tablet_service.cc:1467] Processing CreateTablet for tablet eb6000d7534042f3b00f235a1d803263 (DEFAULT_TABLE table=test-table [id=119f9c0f3d0645219de3c7bcd8e3d57f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:45:06.052609 17755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb6000d7534042f3b00f235a1d803263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:06.061069 17988 tablet_bootstrap.cc:492] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606: Bootstrap starting.
I20260621 02:45:06.064445 17988 tablet_bootstrap.cc:654] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:06.065433 17988 log.cc:824] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606: Log is configured to *not* fsync() on all Append() calls
I20260621 02:45:06.068282 17989 tablet_bootstrap.cc:492] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b: Bootstrap starting.
I20260621 02:45:06.070042 17991 tablet_bootstrap.cc:492] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c: Bootstrap starting.
I20260621 02:45:06.071803 17988 tablet_bootstrap.cc:492] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606: No bootstrap required, opened a new log
I20260621 02:45:06.072054 17988 ts_tablet_manager.cc:1397] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260621 02:45:06.072963 17989 tablet_bootstrap.cc:654] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:06.073604 17991 tablet_bootstrap.cc:654] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:06.074201 17989 log.cc:824] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b: Log is configured to *not* fsync() on all Append() calls
I20260621 02:45:06.074765 17991 log.cc:824] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:45:06.077497 17989 tablet_bootstrap.cc:492] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b: No bootstrap required, opened a new log
I20260621 02:45:06.077754 17989 ts_tablet_manager.cc:1397] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260621 02:45:06.078003 17991 tablet_bootstrap.cc:492] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c: No bootstrap required, opened a new log
I20260621 02:45:06.078253 17991 ts_tablet_manager.cc:1397] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260621 02:45:06.081457 17988 raft_consensus.cc:348] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } }
I20260621 02:45:06.081887 17988 raft_consensus.cc:374] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:06.081986 17988 raft_consensus.cc:729] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8460ee014fce4c72935c507fb0a96606, State: Initialized, Role: FOLLOWER
I20260621 02:45:06.082602 17988 consensus_queue.cc:260] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } }
I20260621 02:45:06.084331 17988 ts_tablet_manager.cc:1428] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260621 02:45:06.086321 17989 raft_consensus.cc:348] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } }
I20260621 02:45:06.086738 17989 raft_consensus.cc:374] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:06.086840 17989 raft_consensus.cc:729] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b20a1a36b827478fb586a00ec935b92b, State: Initialized, Role: FOLLOWER
I20260621 02:45:06.087496 17989 consensus_queue.cc:260] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } }
I20260621 02:45:06.087616 17991 raft_consensus.cc:348] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } }
I20260621 02:45:06.088069 17991 raft_consensus.cc:374] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:06.088171 17991 raft_consensus.cc:729] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf48897dfda844d685b5b2540558361c, State: Initialized, Role: FOLLOWER
I20260621 02:45:06.088689 17991 consensus_queue.cc:260] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } }
I20260621 02:45:06.089227 17964 heartbeater.cc:503] Master 127.9.166.62:38891 was elected leader, sending a full tablet report...
I20260621 02:45:06.091097 17989 ts_tablet_manager.cc:1428] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260621 02:45:06.092007 17825 heartbeater.cc:503] Master 127.9.166.62:38891 was elected leader, sending a full tablet report...
I20260621 02:45:06.092619 17991 ts_tablet_manager.cc:1428] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260621 02:45:06.108078 17996 raft_consensus.cc:482] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:45:06.108395 17996 raft_consensus.cc:504] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } }
I20260621 02:45:06.110186 17996 leader_election.cc:288] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8460ee014fce4c72935c507fb0a96606 (127.9.166.1:32967), b20a1a36b827478fb586a00ec935b92b (127.9.166.3:43225)
I20260621 02:45:06.119057 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.110572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:49237 (local address 127.9.166.1:32967)
0621 02:45:06.110914 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:06.110923 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:06.111165 (+ 242us) server_negotiation.cc:407] Connection header received
0621 02:45:06.111511 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:06.111515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:06.111611 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:06.111712 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:45:06.112413 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.113536 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.116596 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.117802 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.117933 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.118050 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:06.118672 (+ 622us) server_negotiation.cc:299] Negotiation successful
0621 02:45:06.118801 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20260621 02:45:06.119057 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.110995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:59993 (local address 127.9.166.3:43225)
0621 02:45:06.111241 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:06.111246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:06.111566 (+ 320us) server_negotiation.cc:407] Connection header received
0621 02:45:06.111671 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:06.111678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:06.111749 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:06.111833 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:45:06.112578 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.113628 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.116590 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.117801 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.117933 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.118040 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:06.118678 (+ 638us) server_negotiation.cc:299] Negotiation successful
0621 02:45:06.118800 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260621 02:45:06.119329 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.110484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:43225 (local address 127.9.166.2:59993)
0621 02:45:06.111473 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:06.111496 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:06.111579 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:06.111859 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:06.111863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:06.111898 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:45:06.112361 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:06.112373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.113768 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.113778 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:06.116404 (+ 2626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:06.116419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.118114 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.118120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:06.118384 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.118520 (+ 136us) client_negotiation.cc:769] Sending connection context
0621 02:45:06.118628 (+ 108us) client_negotiation.cc:240] Negotiation successful
0621 02:45:06.118837 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":287,"thread_start_us":632,"threads_started":1}
I20260621 02:45:06.119731 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.110320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:32967 (local address 127.9.166.2:49237)
0621 02:45:06.111056 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:06.111077 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:06.111303 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:06.111758 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:06.111761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:06.111779 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:45:06.112253 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:06.112266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.113657 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.113661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:06.116406 (+ 2745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:06.116419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.118120 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.118124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:06.118381 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.118521 (+ 140us) client_negotiation.cc:769] Sending connection context
0621 02:45:06.118621 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 02:45:06.118837 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":303,"thread_start_us":341,"threads_started":1}
I20260621 02:45:06.120337 17918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6000d7534042f3b00f235a1d803263" candidate_uuid: "cf48897dfda844d685b5b2540558361c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b20a1a36b827478fb586a00ec935b92b" is_pre_election: true
I20260621 02:45:06.120874 17918 raft_consensus.cc:2454] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf48897dfda844d685b5b2540558361c in term 0.
I20260621 02:45:06.121418 17640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6000d7534042f3b00f235a1d803263" candidate_uuid: "cf48897dfda844d685b5b2540558361c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8460ee014fce4c72935c507fb0a96606" is_pre_election: true
I20260621 02:45:06.121665 17708 leader_election.cc:302] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b20a1a36b827478fb586a00ec935b92b, cf48897dfda844d685b5b2540558361c; no voters:
I20260621 02:45:06.121949 17640 raft_consensus.cc:2454] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf48897dfda844d685b5b2540558361c in term 0.
I20260621 02:45:06.122164 17996 raft_consensus.cc:2781] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:45:06.122313 17996 raft_consensus.cc:482] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:45:06.122439 17996 raft_consensus.cc:3037] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:06.124668 17996 raft_consensus.cc:504] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } }
I20260621 02:45:06.125272 17996 leader_election.cc:288] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [CANDIDATE]: Term 1 election: Requested vote from peers 8460ee014fce4c72935c507fb0a96606 (127.9.166.1:32967), b20a1a36b827478fb586a00ec935b92b (127.9.166.3:43225)
I20260621 02:45:06.125730 17918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6000d7534042f3b00f235a1d803263" candidate_uuid: "cf48897dfda844d685b5b2540558361c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b20a1a36b827478fb586a00ec935b92b"
I20260621 02:45:06.125728 17640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6000d7534042f3b00f235a1d803263" candidate_uuid: "cf48897dfda844d685b5b2540558361c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8460ee014fce4c72935c507fb0a96606"
I20260621 02:45:06.125936 17640 raft_consensus.cc:3037] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:06.125936 17918 raft_consensus.cc:3037] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Advancing to term 1
W20260621 02:45:06.127553 17826 tablet.cc:2369] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:45:06.129187 17640 raft_consensus.cc:2454] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf48897dfda844d685b5b2540558361c in term 1.
I20260621 02:45:06.129187 17918 raft_consensus.cc:2454] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf48897dfda844d685b5b2540558361c in term 1.
I20260621 02:45:06.129613 17708 leader_election.cc:302] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b20a1a36b827478fb586a00ec935b92b, cf48897dfda844d685b5b2540558361c; no voters:
I20260621 02:45:06.129907 17996 raft_consensus.cc:2781] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:45:06.130345 17996 raft_consensus.cc:686] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [term 1 LEADER]: Becoming Leader. State: Replica: cf48897dfda844d685b5b2540558361c, State: Running, Role: LEADER
W20260621 02:45:06.130787 17687 tablet.cc:2369] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:45:06.130874 17996 consensus_queue.cc:237] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } }
I20260621 02:45:06.135233 17479 catalog_manager.cc:5495] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c reported cstate change: term changed from 0 to 1, leader changed from <none> to cf48897dfda844d685b5b2540558361c (127.9.166.2). New cstate: current_term: 1 leader_uuid: "cf48897dfda844d685b5b2540558361c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } health_report { overall_health: HEALTHY } } }
I20260621 02:45:06.150650 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.145900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.9.166.2:37627)
0621 02:45:06.146072 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:06.146076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:06.146093 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:45:06.146227 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:06.146231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:06.146300 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:06.146414 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:45:06.146724 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.147471 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.148733 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.149125 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.149226 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.149582 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:45:06.150084 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:45:06.150190 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:06.150258 (+ 68us) server_negotiation.cc:299] Negotiation successful
0621 02:45:06.150320 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 02:45:06.158185 17640 raft_consensus.cc:1261] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606 [term 1 FOLLOWER]: Refusing update from remote peer cf48897dfda844d685b5b2540558361c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:45:06.158191 17918 raft_consensus.cc:1261] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b [term 1 FOLLOWER]: Refusing update from remote peer cf48897dfda844d685b5b2540558361c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:45:06.159071 17999 consensus_queue.cc:1035] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:45:06.159427 17996 consensus_queue.cc:1035] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 02:45:06.169711 17965 tablet.cc:2369] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:45:06.172909 18003 mvcc.cc:205] Tried to move back new op lower bound from 7299112575614459904 to 7299112575515860992. Current Snapshot: MvccSnapshot[applied={T|T < 7299112575614459904}]
I20260621 02:45:06.184886 17479 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:45:06.185128 17479 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:45:06.188481 17479 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5: Sending DeleteTablet for 3 replicas of tablet eb6000d7534042f3b00f235a1d803263
I20260621 02:45:06.189538 17894 tablet_service.cc:1514] Processing DeleteTablet for tablet eb6000d7534042f3b00f235a1d803263 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:06 UTC) from {username='oryx', principal='oryx/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:51578
I20260621 02:45:06.189597 17616 tablet_service.cc:1514] Processing DeleteTablet for tablet eb6000d7534042f3b00f235a1d803263 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:06 UTC) from {username='oryx', principal='oryx/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:35826
I20260621 02:45:06.189937 17755 tablet_service.cc:1514] Processing DeleteTablet for tablet eb6000d7534042f3b00f235a1d803263 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:06 UTC) from {username='oryx', principal='oryx/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:51850
I20260621 02:45:06.191991 17479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:45:06.193356 17479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:45:06.203038 17615 tablet_service.cc:1467] Processing CreateTablet for tablet fc6c57ca6a1b4909936ed1cd977f39ab (DEFAULT_TABLE table=test-table [id=21126781295447669981ed090edee242]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:45:06.203598 17615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc6c57ca6a1b4909936ed1cd977f39ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:06.203747 17893 tablet_service.cc:1467] Processing CreateTablet for tablet fc6c57ca6a1b4909936ed1cd977f39ab (DEFAULT_TABLE table=test-table [id=21126781295447669981ed090edee242]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:45:06.204444 17893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc6c57ca6a1b4909936ed1cd977f39ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:06.205919 18014 tablet_replica.cc:330] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c: stopping tablet replica
I20260621 02:45:06.206352 18014 raft_consensus.cc:2229] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:45:06.207019 18014 raft_consensus.cc:2258] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:06.208091 17754 tablet_service.cc:1467] Processing CreateTablet for tablet fc6c57ca6a1b4909936ed1cd977f39ab (DEFAULT_TABLE table=test-table [id=21126781295447669981ed090edee242]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:45:06.208688 17754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc6c57ca6a1b4909936ed1cd977f39ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:06.211123 18012 tablet_replica.cc:330] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b: stopping tablet replica
I20260621 02:45:06.211578 18012 raft_consensus.cc:2229] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:45:06.211648 17988 tablet_bootstrap.cc:492] T fc6c57ca6a1b4909936ed1cd977f39ab P 8460ee014fce4c72935c507fb0a96606: Bootstrap starting.
I20260621 02:45:06.212039 18012 raft_consensus.cc:2258] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:06.213258 17988 tablet_bootstrap.cc:654] T fc6c57ca6a1b4909936ed1cd977f39ab P 8460ee014fce4c72935c507fb0a96606: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:06.215417 18013 tablet_replica.cc:330] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606: stopping tablet replica
I20260621 02:45:06.215899 18013 raft_consensus.cc:2229] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:45:06.216428 18013 raft_consensus.cc:2258] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:06.216909 18014 ts_tablet_manager.cc:1905] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:45:06.218313 18013 ts_tablet_manager.cc:1905] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:45:06.219722 17991 tablet_bootstrap.cc:492] T fc6c57ca6a1b4909936ed1cd977f39ab P cf48897dfda844d685b5b2540558361c: Bootstrap starting.
I20260621 02:45:06.219997 17989 tablet_bootstrap.cc:492] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b: Bootstrap starting.
I20260621 02:45:06.221331 17991 tablet_bootstrap.cc:654] T fc6c57ca6a1b4909936ed1cd977f39ab P cf48897dfda844d685b5b2540558361c: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:06.221573 17989 tablet_bootstrap.cc:654] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:06.221809 18012 ts_tablet_manager.cc:1905] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:45:06.226168 18013 ts_tablet_manager.cc:1918] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:45:06.226361 18013 log.cc:1196] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/eb6000d7534042f3b00f235a1d803263
I20260621 02:45:06.227053 18013 ts_tablet_manager.cc:1939] T eb6000d7534042f3b00f235a1d803263 P 8460ee014fce4c72935c507fb0a96606: Deleting consensus metadata
I20260621 02:45:06.227905 17988 tablet_bootstrap.cc:492] T fc6c57ca6a1b4909936ed1cd977f39ab P 8460ee014fce4c72935c507fb0a96606: No bootstrap required, opened a new log
I20260621 02:45:06.228081 17988 ts_tablet_manager.cc:1397] T fc6c57ca6a1b4909936ed1cd977f39ab P 8460ee014fce4c72935c507fb0a96606: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20260621 02:45:06.229048 17463 catalog_manager.cc:4841] TS 8460ee014fce4c72935c507fb0a96606 (127.9.166.1:32967): tablet eb6000d7534042f3b00f235a1d803263 (table test-table [id=119f9c0f3d0645219de3c7bcd8e3d57f]) successfully deleted
I20260621 02:45:06.229041 17988 raft_consensus.cc:348] T fc6c57ca6a1b4909936ed1cd977f39ab P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.229264 17988 raft_consensus.cc:374] T fc6c57ca6a1b4909936ed1cd977f39ab P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:06.229390 17988 raft_consensus.cc:729] T fc6c57ca6a1b4909936ed1cd977f39ab P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8460ee014fce4c72935c507fb0a96606, State: Initialized, Role: FOLLOWER
I20260621 02:45:06.229720 17988 consensus_queue.cc:260] T fc6c57ca6a1b4909936ed1cd977f39ab P 8460ee014fce4c72935c507fb0a96606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.230460 17988 ts_tablet_manager.cc:1428] T fc6c57ca6a1b4909936ed1cd977f39ab P 8460ee014fce4c72935c507fb0a96606: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260621 02:45:06.235899 18012 ts_tablet_manager.cc:1918] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:45:06.236076 18012 log.cc:1196] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/eb6000d7534042f3b00f235a1d803263
I20260621 02:45:06.236729 18012 ts_tablet_manager.cc:1939] T eb6000d7534042f3b00f235a1d803263 P b20a1a36b827478fb586a00ec935b92b: Deleting consensus metadata
I20260621 02:45:06.238691 17465 catalog_manager.cc:4841] TS b20a1a36b827478fb586a00ec935b92b (127.9.166.3:43225): tablet eb6000d7534042f3b00f235a1d803263 (table test-table [id=119f9c0f3d0645219de3c7bcd8e3d57f]) successfully deleted
I20260621 02:45:06.238796 17989 tablet_bootstrap.cc:492] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b: No bootstrap required, opened a new log
I20260621 02:45:06.238808 17991 tablet_bootstrap.cc:492] T fc6c57ca6a1b4909936ed1cd977f39ab P cf48897dfda844d685b5b2540558361c: No bootstrap required, opened a new log
I20260621 02:45:06.238947 17989 ts_tablet_manager.cc:1397] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.007s
I20260621 02:45:06.238947 17991 ts_tablet_manager.cc:1397] T fc6c57ca6a1b4909936ed1cd977f39ab P cf48897dfda844d685b5b2540558361c: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.005s
I20260621 02:45:06.239289 18014 ts_tablet_manager.cc:1918] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:45:06.239516 18014 log.cc:1196] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/eb6000d7534042f3b00f235a1d803263
I20260621 02:45:06.239778 17989 raft_consensus.cc:348] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.239801 17991 raft_consensus.cc:348] T fc6c57ca6a1b4909936ed1cd977f39ab P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.239985 18014 ts_tablet_manager.cc:1939] T eb6000d7534042f3b00f235a1d803263 P cf48897dfda844d685b5b2540558361c: Deleting consensus metadata
I20260621 02:45:06.239998 17989 raft_consensus.cc:374] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:06.240016 17991 raft_consensus.cc:374] T fc6c57ca6a1b4909936ed1cd977f39ab P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:06.240091 17989 raft_consensus.cc:729] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b20a1a36b827478fb586a00ec935b92b, State: Initialized, Role: FOLLOWER
I20260621 02:45:06.240109 17991 raft_consensus.cc:729] T fc6c57ca6a1b4909936ed1cd977f39ab P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf48897dfda844d685b5b2540558361c, State: Initialized, Role: FOLLOWER
I20260621 02:45:06.240377 17991 consensus_queue.cc:260] T fc6c57ca6a1b4909936ed1cd977f39ab P cf48897dfda844d685b5b2540558361c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.240375 17989 consensus_queue.cc:260] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.240917 17991 ts_tablet_manager.cc:1428] T fc6c57ca6a1b4909936ed1cd977f39ab P cf48897dfda844d685b5b2540558361c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:45:06.241587 17989 ts_tablet_manager.cc:1428] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260621 02:45:06.248410 17463 catalog_manager.cc:4841] TS cf48897dfda844d685b5b2540558361c (127.9.166.2:37627): tablet eb6000d7534042f3b00f235a1d803263 (table test-table [id=119f9c0f3d0645219de3c7bcd8e3d57f]) successfully deleted
I20260621 02:45:06.258710 17995 raft_consensus.cc:482] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:45:06.258934 17995 raft_consensus.cc:504] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.260918 17995 leader_election.cc:288] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8460ee014fce4c72935c507fb0a96606 (127.9.166.1:32967), cf48897dfda844d685b5b2540558361c (127.9.166.2:37627)
I20260621 02:45:06.268268 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.261100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:32967 (local address 127.9.166.3:40215)
0621 02:45:06.261284 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:06.261304 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:06.261423 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:06.261857 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:06.261861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:06.261894 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:45:06.262125 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:06.262145 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.263626 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.263631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:06.265714 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:06.265733 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.267487 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.267493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:06.267693 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.267728 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 02:45:06.267822 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 02:45:06.267951 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":39}
I20260621 02:45:06.268949 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.261294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:40215 (local address 127.9.166.1:32967)
0621 02:45:06.261459 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:06.261468 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:06.261489 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:45:06.261658 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:06.261664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:06.261743 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:06.261841 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:45:06.262319 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.263466 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.265908 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.267158 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.267277 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.267377 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:06.268558 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0621 02:45:06.268692 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:45:06.269491 17640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6c57ca6a1b4909936ed1cd977f39ab" candidate_uuid: "b20a1a36b827478fb586a00ec935b92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8460ee014fce4c72935c507fb0a96606" is_pre_election: true
I20260621 02:45:06.269767 17640 raft_consensus.cc:2454] T fc6c57ca6a1b4909936ed1cd977f39ab P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b20a1a36b827478fb586a00ec935b92b in term 0.
I20260621 02:45:06.270066 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.261845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:33895 (local address 127.9.166.2:37627)
0621 02:45:06.262028 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:06.262033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:06.262615 (+ 582us) server_negotiation.cc:407] Connection header received
0621 02:45:06.262742 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:06.262747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:06.262815 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:06.262898 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:45:06.263597 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.264849 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.267267 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.268590 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.268702 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.268787 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:06.269582 (+ 795us) server_negotiation.cc:299] Negotiation successful
0621 02:45:06.269704 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260621 02:45:06.270298 17845 leader_election.cc:302] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8460ee014fce4c72935c507fb0a96606, b20a1a36b827478fb586a00ec935b92b; no voters:
I20260621 02:45:06.270709 17995 raft_consensus.cc:2781] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:45:06.270849 17995 raft_consensus.cc:482] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:45:06.270962 17995 raft_consensus.cc:3037] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:06.271186 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.261408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:37627 (local address 127.9.166.3:33895)
0621 02:45:06.262499 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:06.262516 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:06.262639 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:06.263069 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:06.263074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:06.263098 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:45:06.263440 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:06.263450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.265026 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.265031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:06.267107 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:06.267121 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.269218 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.269225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:06.269393 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.269430 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 02:45:06.270549 (+ 1119us) client_negotiation.cc:240] Negotiation successful
0621 02:45:06.270687 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":27,"thread_start_us":782,"threads_started":1}
I20260621 02:45:06.274091 17995 raft_consensus.cc:504] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.274784 17995 leader_election.cc:288] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [CANDIDATE]: Term 1 election: Requested vote from peers 8460ee014fce4c72935c507fb0a96606 (127.9.166.1:32967), cf48897dfda844d685b5b2540558361c (127.9.166.2:37627)
I20260621 02:45:06.276561 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6c57ca6a1b4909936ed1cd977f39ab" candidate_uuid: "b20a1a36b827478fb586a00ec935b92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf48897dfda844d685b5b2540558361c" is_pre_election: true
I20260621 02:45:06.277613 17775 raft_consensus.cc:2454] T fc6c57ca6a1b4909936ed1cd977f39ab P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b20a1a36b827478fb586a00ec935b92b in term 0.
I20260621 02:45:06.277740 17640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6c57ca6a1b4909936ed1cd977f39ab" candidate_uuid: "b20a1a36b827478fb586a00ec935b92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8460ee014fce4c72935c507fb0a96606"
I20260621 02:45:06.277942 17640 raft_consensus.cc:3037] T fc6c57ca6a1b4909936ed1cd977f39ab P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:06.280926 17640 raft_consensus.cc:2454] T fc6c57ca6a1b4909936ed1cd977f39ab P 8460ee014fce4c72935c507fb0a96606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b20a1a36b827478fb586a00ec935b92b in term 1.
I20260621 02:45:06.281368 17845 leader_election.cc:302] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8460ee014fce4c72935c507fb0a96606, b20a1a36b827478fb586a00ec935b92b; no voters:
I20260621 02:45:06.281688 17995 raft_consensus.cc:2781] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:45:06.282199 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6c57ca6a1b4909936ed1cd977f39ab" candidate_uuid: "b20a1a36b827478fb586a00ec935b92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf48897dfda844d685b5b2540558361c"
I20260621 02:45:06.282402 17775 raft_consensus.cc:3037] T fc6c57ca6a1b4909936ed1cd977f39ab P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:06.285861 17775 raft_consensus.cc:2454] T fc6c57ca6a1b4909936ed1cd977f39ab P cf48897dfda844d685b5b2540558361c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b20a1a36b827478fb586a00ec935b92b in term 1.
I20260621 02:45:06.290292 17995 raft_consensus.cc:686] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [term 1 LEADER]: Becoming Leader. State: Replica: b20a1a36b827478fb586a00ec935b92b, State: Running, Role: LEADER
I20260621 02:45:06.290745 17995 consensus_queue.cc:237] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.294836 17478 catalog_manager.cc:5495] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b reported cstate change: term changed from 0 to 1, leader changed from <none> to b20a1a36b827478fb586a00ec935b92b (127.9.166.3). New cstate: current_term: 1 leader_uuid: "b20a1a36b827478fb586a00ec935b92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } health_report { overall_health: HEALTHY } } }
Jun 21 02:45:06 dist-test-slave-hvgn krb5kdc[17434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009904, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.62@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM
I20260621 02:45:06.344568 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.329861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38891 (local address 127.0.0.1:40994)
0621 02:45:06.330514 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:06.330535 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:06.330670 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:06.331013 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:06.331016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:06.331792 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:06.332047 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:06.332055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.333475 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.333478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:06.334422 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:06.334431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.335328 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.335331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:06.335503 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.336186 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:06.336203 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:06.338451 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:45:06.342051 (+ 3600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:06.342063 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:06.342219 (+ 156us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:06.342692 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:06.343089 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:06.343092 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:06.343094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:06.343226 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:06.343794 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:45:06.343799 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:45:06.343944 (+ 145us) client_negotiation.cc:769] Sending connection context
0621 02:45:06.344081 (+ 137us) client_negotiation.cc:240] Negotiation successful
0621 02:45:06.344255 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":385,"threads_started":1}
I20260621 02:45:06.344851 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.329855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40994 (local address 127.9.166.62:38891)
0621 02:45:06.330064 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:06.330073 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:06.330633 (+ 560us) server_negotiation.cc:407] Connection header received
0621 02:45:06.330780 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:06.330786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:06.330864 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:06.330972 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:06.332177 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.333344 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.334549 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.335166 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.335287 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.338622 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:06.338629 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:06.338634 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:06.338670 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:06.341923 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:06.342812 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:06.342818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:06.342821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:06.342965 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:06.343324 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:06.343329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:06.343333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:06.343673 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:06.343835 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:06.344202 (+ 367us) server_negotiation.cc:299] Negotiation successful
0621 02:45:06.344420 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 02:45:06.350083 17478 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:40994:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260621 02:45:06.351212 17478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:45:06.357918 17616 tablet_service.cc:1467] Processing CreateTablet for tablet 6b2065a3c4834715bef6ccffb3c61e62 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=413375587c15437da556509aadca49f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:45:06.358103 17755 tablet_service.cc:1467] Processing CreateTablet for tablet 6b2065a3c4834715bef6ccffb3c61e62 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=413375587c15437da556509aadca49f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:45:06.358323 17893 tablet_service.cc:1467] Processing CreateTablet for tablet 6b2065a3c4834715bef6ccffb3c61e62 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=413375587c15437da556509aadca49f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260621 02:45:06.358515 17616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b2065a3c4834715bef6ccffb3c61e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:06.358647 17755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b2065a3c4834715bef6ccffb3c61e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:06.358800 17893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b2065a3c4834715bef6ccffb3c61e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:06.365818 17989 tablet_bootstrap.cc:492] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b: Bootstrap starting.
I20260621 02:45:06.366544 17991 tablet_bootstrap.cc:492] T 6b2065a3c4834715bef6ccffb3c61e62 P cf48897dfda844d685b5b2540558361c: Bootstrap starting.
I20260621 02:45:06.366631 17988 tablet_bootstrap.cc:492] T 6b2065a3c4834715bef6ccffb3c61e62 P 8460ee014fce4c72935c507fb0a96606: Bootstrap starting.
I20260621 02:45:06.367522 17989 tablet_bootstrap.cc:654] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:06.368252 17988 tablet_bootstrap.cc:654] T 6b2065a3c4834715bef6ccffb3c61e62 P 8460ee014fce4c72935c507fb0a96606: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:06.368307 17991 tablet_bootstrap.cc:654] T 6b2065a3c4834715bef6ccffb3c61e62 P cf48897dfda844d685b5b2540558361c: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:06.379218 17991 tablet_bootstrap.cc:492] T 6b2065a3c4834715bef6ccffb3c61e62 P cf48897dfda844d685b5b2540558361c: No bootstrap required, opened a new log
I20260621 02:45:06.379506 17991 ts_tablet_manager.cc:1397] T 6b2065a3c4834715bef6ccffb3c61e62 P cf48897dfda844d685b5b2540558361c: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260621 02:45:06.379532 17989 tablet_bootstrap.cc:492] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b: No bootstrap required, opened a new log
I20260621 02:45:06.379689 17989 ts_tablet_manager.cc:1397] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20260621 02:45:06.380414 17989 raft_consensus.cc:348] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.380667 17989 raft_consensus.cc:374] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:06.380745 17989 raft_consensus.cc:729] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b20a1a36b827478fb586a00ec935b92b, State: Initialized, Role: FOLLOWER
I20260621 02:45:06.380475 17991 raft_consensus.cc:348] T 6b2065a3c4834715bef6ccffb3c61e62 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.380940 17991 raft_consensus.cc:374] T 6b2065a3c4834715bef6ccffb3c61e62 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:06.381075 17991 raft_consensus.cc:729] T 6b2065a3c4834715bef6ccffb3c61e62 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf48897dfda844d685b5b2540558361c, State: Initialized, Role: FOLLOWER
I20260621 02:45:06.381029 17989 consensus_queue.cc:260] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.381435 17995 tablet_replica.cc:439] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } } }
I20260621 02:45:06.381640 17989 ts_tablet_manager.cc:1428] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260621 02:45:06.381742 17995 tablet_replica.cc:442] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:45:06.382529 17991 consensus_queue.cc:260] T 6b2065a3c4834715bef6ccffb3c61e62 P cf48897dfda844d685b5b2540558361c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.383257 17991 ts_tablet_manager.cc:1428] T 6b2065a3c4834715bef6ccffb3c61e62 P cf48897dfda844d685b5b2540558361c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260621 02:45:06.383067 18007 tablet_replica.cc:439] T 6b2065a3c4834715bef6ccffb3c61e62 P cf48897dfda844d685b5b2540558361c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } } }
I20260621 02:45:06.384092 18007 tablet_replica.cc:442] T 6b2065a3c4834715bef6ccffb3c61e62 P cf48897dfda844d685b5b2540558361c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:45:06.385092 17988 tablet_bootstrap.cc:492] T 6b2065a3c4834715bef6ccffb3c61e62 P 8460ee014fce4c72935c507fb0a96606: No bootstrap required, opened a new log
I20260621 02:45:06.385262 17988 ts_tablet_manager.cc:1397] T 6b2065a3c4834715bef6ccffb3c61e62 P 8460ee014fce4c72935c507fb0a96606: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.005s
I20260621 02:45:06.385994 17988 raft_consensus.cc:348] T 6b2065a3c4834715bef6ccffb3c61e62 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.386256 17988 raft_consensus.cc:374] T 6b2065a3c4834715bef6ccffb3c61e62 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:06.386346 17988 raft_consensus.cc:729] T 6b2065a3c4834715bef6ccffb3c61e62 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8460ee014fce4c72935c507fb0a96606, State: Initialized, Role: FOLLOWER
I20260621 02:45:06.386624 17988 consensus_queue.cc:260] T 6b2065a3c4834715bef6ccffb3c61e62 P 8460ee014fce4c72935c507fb0a96606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.387038 17994 tablet_replica.cc:439] T 6b2065a3c4834715bef6ccffb3c61e62 P 8460ee014fce4c72935c507fb0a96606: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } } }
I20260621 02:45:06.387372 17994 tablet_replica.cc:442] T 6b2065a3c4834715bef6ccffb3c61e62 P 8460ee014fce4c72935c507fb0a96606: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:45:06.388123 17988 ts_tablet_manager.cc:1428] T 6b2065a3c4834715bef6ccffb3c61e62 P 8460ee014fce4c72935c507fb0a96606: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260621 02:45:06.551760 17995 raft_consensus.cc:482] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:45:06.552013 17995 raft_consensus.cc:504] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.552809 17995 leader_election.cc:288] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8460ee014fce4c72935c507fb0a96606 (127.9.166.1:32967), cf48897dfda844d685b5b2540558361c (127.9.166.2:37627)
I20260621 02:45:06.553547 17640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2065a3c4834715bef6ccffb3c61e62" candidate_uuid: "b20a1a36b827478fb586a00ec935b92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8460ee014fce4c72935c507fb0a96606" is_pre_election: true
I20260621 02:45:06.553553 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2065a3c4834715bef6ccffb3c61e62" candidate_uuid: "b20a1a36b827478fb586a00ec935b92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf48897dfda844d685b5b2540558361c" is_pre_election: true
I20260621 02:45:06.553812 17640 raft_consensus.cc:2454] T 6b2065a3c4834715bef6ccffb3c61e62 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b20a1a36b827478fb586a00ec935b92b in term 0.
I20260621 02:45:06.553820 17775 raft_consensus.cc:2454] T 6b2065a3c4834715bef6ccffb3c61e62 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b20a1a36b827478fb586a00ec935b92b in term 0.
I20260621 02:45:06.554293 17845 leader_election.cc:302] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b20a1a36b827478fb586a00ec935b92b, cf48897dfda844d685b5b2540558361c; no voters:
I20260621 02:45:06.554592 17995 raft_consensus.cc:2781] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:45:06.554721 17995 raft_consensus.cc:482] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:45:06.554858 17995 raft_consensus.cc:3037] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:06.556880 17995 raft_consensus.cc:504] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.557427 17995 leader_election.cc:288] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b [CANDIDATE]: Term 1 election: Requested vote from peers 8460ee014fce4c72935c507fb0a96606 (127.9.166.1:32967), cf48897dfda844d685b5b2540558361c (127.9.166.2:37627)
I20260621 02:45:06.557881 17640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2065a3c4834715bef6ccffb3c61e62" candidate_uuid: "b20a1a36b827478fb586a00ec935b92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8460ee014fce4c72935c507fb0a96606"
I20260621 02:45:06.557950 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2065a3c4834715bef6ccffb3c61e62" candidate_uuid: "b20a1a36b827478fb586a00ec935b92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf48897dfda844d685b5b2540558361c"
I20260621 02:45:06.558127 17640 raft_consensus.cc:3037] T 6b2065a3c4834715bef6ccffb3c61e62 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:06.558163 17775 raft_consensus.cc:3037] T 6b2065a3c4834715bef6ccffb3c61e62 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:06.560354 17775 raft_consensus.cc:2454] T 6b2065a3c4834715bef6ccffb3c61e62 P cf48897dfda844d685b5b2540558361c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b20a1a36b827478fb586a00ec935b92b in term 1.
I20260621 02:45:06.560717 17845 leader_election.cc:302] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b20a1a36b827478fb586a00ec935b92b, cf48897dfda844d685b5b2540558361c; no voters:
I20260621 02:45:06.560988 17995 raft_consensus.cc:2781] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:45:06.561165 17995 raft_consensus.cc:686] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b [term 1 LEADER]: Becoming Leader. State: Replica: b20a1a36b827478fb586a00ec935b92b, State: Running, Role: LEADER
I20260621 02:45:06.561414 17995 consensus_queue.cc:237] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.561856 17640 raft_consensus.cc:2454] T 6b2065a3c4834715bef6ccffb3c61e62 P 8460ee014fce4c72935c507fb0a96606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b20a1a36b827478fb586a00ec935b92b in term 1.
I20260621 02:45:06.563033 18019 tablet_replica.cc:439] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b: TxnStatusTablet state changed. Reason: New leader b20a1a36b827478fb586a00ec935b92b. Latest consensus state: current_term: 1 leader_uuid: "b20a1a36b827478fb586a00ec935b92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } } }
I20260621 02:45:06.563293 18019 tablet_replica.cc:442] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:45:06.565183 17478 catalog_manager.cc:5495] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b reported cstate change: term changed from 0 to 1, leader changed from <none> to b20a1a36b827478fb586a00ec935b92b (127.9.166.3). New cstate: current_term: 1 leader_uuid: "b20a1a36b827478fb586a00ec935b92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } health_report { overall_health: HEALTHY } } }
I20260621 02:45:06.567842 18031 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260621 02:45:06.568091 18031 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260621 02:45:06.625758 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.618156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:43225 (local address 127.0.0.1:51584)
0621 02:45:06.618476 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:06.618494 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:06.618635 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:06.619130 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:06.619142 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:06.619155 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:45:06.619344 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:06.619353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.620912 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.620918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:06.622617 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:06.622631 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.623817 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.623823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:06.624011 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.624076 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:45:06.625148 (+ 1072us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:45:06.625166 (+ 18us) client_negotiation.cc:769] Sending connection context
0621 02:45:06.625257 (+ 91us) client_negotiation.cc:240] Negotiation successful
0621 02:45:06.625409 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260621 02:45:06.626377 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.618225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51584 (local address 127.9.166.3:43225)
0621 02:45:06.618433 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:06.618440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:06.618570 (+ 130us) server_negotiation.cc:407] Connection header received
0621 02:45:06.618838 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:06.618843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:06.618940 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:06.619099 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:45:06.619483 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.620748 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.622854 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.623570 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.623687 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.624294 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:45:06.624950 (+ 656us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:45:06.625093 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:06.625979 (+ 886us) server_negotiation.cc:299] Negotiation successful
0621 02:45:06.626056 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:45:06.630532 17640 raft_consensus.cc:1261] T 6b2065a3c4834715bef6ccffb3c61e62 P 8460ee014fce4c72935c507fb0a96606 [term 1 FOLLOWER]: Refusing update from remote peer b20a1a36b827478fb586a00ec935b92b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:45:06.630635 17775 raft_consensus.cc:1261] T 6b2065a3c4834715bef6ccffb3c61e62 P cf48897dfda844d685b5b2540558361c [term 1 FOLLOWER]: Refusing update from remote peer b20a1a36b827478fb586a00ec935b92b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:45:06.631323 18019 consensus_queue.cc:1035] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:45:06.631758 17995 consensus_queue.cc:1035] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:45:06.644367 17994 tablet_replica.cc:439] T 6b2065a3c4834715bef6ccffb3c61e62 P 8460ee014fce4c72935c507fb0a96606: TxnStatusTablet state changed. Reason: New leader b20a1a36b827478fb586a00ec935b92b. Latest consensus state: current_term: 1 leader_uuid: "b20a1a36b827478fb586a00ec935b92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } } }
I20260621 02:45:06.644744 17994 tablet_replica.cc:442] T 6b2065a3c4834715bef6ccffb3c61e62 P 8460ee014fce4c72935c507fb0a96606: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:45:06.649628 18019 tablet_replica.cc:439] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b20a1a36b827478fb586a00ec935b92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } } }
I20260621 02:45:06.649933 18019 tablet_replica.cc:442] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:45:06.650254 18004 mvcc.cc:205] Tried to move back new op lower bound from 7299112577550798848 to 7299112577277624320. Current Snapshot: MvccSnapshot[applied={T|T < 7299112577550798848 or (T in {7299112577550798848})}]
I20260621 02:45:06.658321 17994 tablet_replica.cc:439] T 6b2065a3c4834715bef6ccffb3c61e62 P 8460ee014fce4c72935c507fb0a96606: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b20a1a36b827478fb586a00ec935b92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } } }
I20260621 02:45:06.658643 17994 tablet_replica.cc:442] T 6b2065a3c4834715bef6ccffb3c61e62 P 8460ee014fce4c72935c507fb0a96606: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:45:06.671324 18007 tablet_replica.cc:439] T 6b2065a3c4834715bef6ccffb3c61e62 P cf48897dfda844d685b5b2540558361c: TxnStatusTablet state changed. Reason: New leader b20a1a36b827478fb586a00ec935b92b. Latest consensus state: current_term: 1 leader_uuid: "b20a1a36b827478fb586a00ec935b92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } } }
I20260621 02:45:06.671608 18007 tablet_replica.cc:442] T 6b2065a3c4834715bef6ccffb3c61e62 P cf48897dfda844d685b5b2540558361c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:45:06.673918 17995 tablet_replica.cc:439] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b20a1a36b827478fb586a00ec935b92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } } }
I20260621 02:45:06.674209 17995 tablet_replica.cc:442] T 6b2065a3c4834715bef6ccffb3c61e62 P b20a1a36b827478fb586a00ec935b92b: This TxnStatusTablet replica's current role is: LEADER
I20260621 02:45:06.675810 18002 mvcc.cc:205] Tried to move back new op lower bound from 7299112577550798848 to 7299112577277624320. Current Snapshot: MvccSnapshot[applied={T|T < 7299112577550798848}]
I20260621 02:45:06.677678 18007 tablet_replica.cc:439] T 6b2065a3c4834715bef6ccffb3c61e62 P cf48897dfda844d685b5b2540558361c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b20a1a36b827478fb586a00ec935b92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } } }
I20260621 02:45:06.677917 18007 tablet_replica.cc:442] T 6b2065a3c4834715bef6ccffb3c61e62 P cf48897dfda844d685b5b2540558361c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260621 02:45:06.678207 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.670362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.9.166.3:43225)
0621 02:45:06.670551 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:06.670557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:06.671237 (+ 680us) server_negotiation.cc:407] Connection header received
0621 02:45:06.671562 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:06.671567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:06.671688 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:06.671794 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:45:06.672513 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.673681 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.675983 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.676697 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.676802 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.677417 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:45:06.677552 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:45:06.677613 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:06.677844 (+ 231us) server_negotiation.cc:299] Negotiation successful
0621 02:45:06.677921 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:45:06.695302 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.688864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:47271 (local address 127.9.166.3:43225)
0621 02:45:06.689035 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:06.689039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:06.689665 (+ 626us) server_negotiation.cc:407] Connection header received
0621 02:45:06.689831 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:06.689837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:06.689914 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:06.690012 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:45:06.690502 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.691753 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.693184 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.693931 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.694058 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.694487 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:45:06.694637 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:45:06.694737 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:06.694858 (+ 121us) server_negotiation.cc:299] Negotiation successful
0621 02:45:06.694935 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":48}
I20260621 02:45:06.695322 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.688942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:43225 (local address 127.9.166.3:47271)
0621 02:45:06.689570 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:06.689585 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:06.689699 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:06.690048 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:06.690051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:06.690060 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0621 02:45:06.690354 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:06.690364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.691895 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.691899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:06.693047 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:06.693057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.694133 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.694139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:06.694259 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.694314 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0621 02:45:06.694741 (+ 427us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0621 02:45:06.694750 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 02:45:06.694815 (+ 65us) client_negotiation.cc:240] Negotiation successful
0621 02:45:06.694993 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":349,"threads_started":1}
I20260621 02:45:06.705549 17775 raft_consensus.cc:1261] T fc6c57ca6a1b4909936ed1cd977f39ab P cf48897dfda844d685b5b2540558361c [term 1 FOLLOWER]: Refusing update from remote peer b20a1a36b827478fb586a00ec935b92b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:45:06.705587 17640 raft_consensus.cc:1261] T fc6c57ca6a1b4909936ed1cd977f39ab P 8460ee014fce4c72935c507fb0a96606 [term 1 FOLLOWER]: Refusing update from remote peer b20a1a36b827478fb586a00ec935b92b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:45:06.706066 18019 consensus_queue.cc:1035] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:45:06.706415 18042 consensus_queue.cc:1035] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:45:06.775539 17478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:45:06.775787 17478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:45:06.778708 17478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5: Sending DeleteTablet for 3 replicas of tablet fc6c57ca6a1b4909936ed1cd977f39ab
I20260621 02:45:06.779613 17893 tablet_service.cc:1514] Processing DeleteTablet for tablet fc6c57ca6a1b4909936ed1cd977f39ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:06 UTC) from {username='oryx', principal='oryx/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:51578
I20260621 02:45:06.779951 17616 tablet_service.cc:1514] Processing DeleteTablet for tablet fc6c57ca6a1b4909936ed1cd977f39ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:06 UTC) from {username='oryx', principal='oryx/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:35826
I20260621 02:45:06.779965 17755 tablet_service.cc:1514] Processing DeleteTablet for tablet fc6c57ca6a1b4909936ed1cd977f39ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:06 UTC) from {username='oryx', principal='oryx/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:51850
I20260621 02:45:06.792284 18050 tablet_replica.cc:330] T fc6c57ca6a1b4909936ed1cd977f39ab P cf48897dfda844d685b5b2540558361c: stopping tablet replica
I20260621 02:45:06.792596 18050 raft_consensus.cc:2229] T fc6c57ca6a1b4909936ed1cd977f39ab P cf48897dfda844d685b5b2540558361c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:45:06.792887 18050 raft_consensus.cc:2258] T fc6c57ca6a1b4909936ed1cd977f39ab P cf48897dfda844d685b5b2540558361c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:06.793537 18048 tablet_replica.cc:330] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b: stopping tablet replica
I20260621 02:45:06.793746 18048 raft_consensus.cc:2229] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:45:06.794286 18050 ts_tablet_manager.cc:1905] T fc6c57ca6a1b4909936ed1cd977f39ab P cf48897dfda844d685b5b2540558361c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:45:06.794344 18048 raft_consensus.cc:2258] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:06.795324 18048 ts_tablet_manager.cc:1905] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:45:06.802287 18050 ts_tablet_manager.cc:1918] T fc6c57ca6a1b4909936ed1cd977f39ab P cf48897dfda844d685b5b2540558361c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:45:06.802313 18048 ts_tablet_manager.cc:1918] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:45:06.802438 18050 log.cc:1196] T fc6c57ca6a1b4909936ed1cd977f39ab P cf48897dfda844d685b5b2540558361c: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/fc6c57ca6a1b4909936ed1cd977f39ab
I20260621 02:45:06.802444 18048 log.cc:1196] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/fc6c57ca6a1b4909936ed1cd977f39ab
I20260621 02:45:06.802919 18048 ts_tablet_manager.cc:1939] T fc6c57ca6a1b4909936ed1cd977f39ab P b20a1a36b827478fb586a00ec935b92b: Deleting consensus metadata
I20260621 02:45:06.802913 18050 ts_tablet_manager.cc:1939] T fc6c57ca6a1b4909936ed1cd977f39ab P cf48897dfda844d685b5b2540558361c: Deleting consensus metadata
I20260621 02:45:06.803315 18049 tablet_replica.cc:330] T fc6c57ca6a1b4909936ed1cd977f39ab P 8460ee014fce4c72935c507fb0a96606: stopping tablet replica
I20260621 02:45:06.803704 18049 raft_consensus.cc:2229] T fc6c57ca6a1b4909936ed1cd977f39ab P 8460ee014fce4c72935c507fb0a96606 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:45:06.804055 17465 catalog_manager.cc:4841] TS b20a1a36b827478fb586a00ec935b92b (127.9.166.3:43225): tablet fc6c57ca6a1b4909936ed1cd977f39ab (table test-table [id=21126781295447669981ed090edee242]) successfully deleted
I20260621 02:45:06.804059 17463 catalog_manager.cc:4841] TS cf48897dfda844d685b5b2540558361c (127.9.166.2:37627): tablet fc6c57ca6a1b4909936ed1cd977f39ab (table test-table [id=21126781295447669981ed090edee242]) successfully deleted
I20260621 02:45:06.804410 18049 raft_consensus.cc:2258] T fc6c57ca6a1b4909936ed1cd977f39ab P 8460ee014fce4c72935c507fb0a96606 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:06.808139 18049 ts_tablet_manager.cc:1905] T fc6c57ca6a1b4909936ed1cd977f39ab P 8460ee014fce4c72935c507fb0a96606: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:45:06.813341 18049 ts_tablet_manager.cc:1918] T fc6c57ca6a1b4909936ed1cd977f39ab P 8460ee014fce4c72935c507fb0a96606: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:45:06.813468 18049 log.cc:1196] T fc6c57ca6a1b4909936ed1cd977f39ab P 8460ee014fce4c72935c507fb0a96606: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/fc6c57ca6a1b4909936ed1cd977f39ab
I20260621 02:45:06.813858 18049 ts_tablet_manager.cc:1939] T fc6c57ca6a1b4909936ed1cd977f39ab P 8460ee014fce4c72935c507fb0a96606: Deleting consensus metadata
I20260621 02:45:06.814767 17463 catalog_manager.cc:4841] TS 8460ee014fce4c72935c507fb0a96606 (127.9.166.1:32967): tablet fc6c57ca6a1b4909936ed1cd977f39ab (table test-table [id=21126781295447669981ed090edee242]) successfully deleted
Jun 21 02:45:06 dist-test-slave-hvgn krb5kdc[17434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 21 02:45:06 dist-test-slave-hvgn krb5kdc[17434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:06 dist-test-slave-hvgn krb5kdc[17434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 02:45:06.857787 17574 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:06.842147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40996 (local address 127.9.166.62:38891)
0621 02:45:06.842414 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:06.842422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:06.842706 (+ 284us) server_negotiation.cc:407] Connection header received
0621 02:45:06.842905 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:06.842909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:06.842975 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:06.843054 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:06.844699 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.845415 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.847170 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.847638 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.847734 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.857521 (+ 9787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40996: BlockingRecv error: recv got EOF from 127.0.0.1:40996 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 21 02:45:06 dist-test-slave-hvgn krb5kdc[17434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM
I20260621 02:45:06.876380 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.861192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41008 (local address 127.9.166.62:38891)
0621 02:45:06.861390 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:06.861397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:06.861732 (+ 335us) server_negotiation.cc:407] Connection header received
0621 02:45:06.861921 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:06.861925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:06.861987 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:06.862065 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:06.863604 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.864322 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.866113 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.866500 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.866589 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.870769 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:06.870774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:06.870777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:06.870809 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:06.873524 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:06.874158 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:06.874168 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:06.874170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:06.874361 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:06.874838 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:06.874843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:06.874853 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:06.875166 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:06.875448 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:06.875784 (+ 336us) server_negotiation.cc:299] Negotiation successful
0621 02:45:06.876029 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 02:45:06.880319 17478 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:45:06.881100 17478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:45:06.889030 17616 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3ffcf3209a417d825a5b17a800a4e9 (DEFAULT_TABLE table=test-table [id=83cb0881353d42b0a35151bb3539fc8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:45:06.889482 17616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a3ffcf3209a417d825a5b17a800a4e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:06.893990 17893 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3ffcf3209a417d825a5b17a800a4e9 (DEFAULT_TABLE table=test-table [id=83cb0881353d42b0a35151bb3539fc8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:45:06.894434 17893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a3ffcf3209a417d825a5b17a800a4e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:06.896212 17755 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3ffcf3209a417d825a5b17a800a4e9 (DEFAULT_TABLE table=test-table [id=83cb0881353d42b0a35151bb3539fc8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:45:06.896565 17755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a3ffcf3209a417d825a5b17a800a4e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:06.904109 18063 tablet_bootstrap.cc:492] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606: Bootstrap starting.
I20260621 02:45:06.905932 18063 tablet_bootstrap.cc:654] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:06.908771 18063 tablet_bootstrap.cc:492] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606: No bootstrap required, opened a new log
I20260621 02:45:06.908948 18063 ts_tablet_manager.cc:1397] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260621 02:45:06.909792 18063 raft_consensus.cc:348] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.910055 18063 raft_consensus.cc:374] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:06.910151 18063 raft_consensus.cc:729] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8460ee014fce4c72935c507fb0a96606, State: Initialized, Role: FOLLOWER
I20260621 02:45:06.910485 18063 consensus_queue.cc:260] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.911727 18063 ts_tablet_manager.cc:1428] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260621 02:45:06.914388 18066 tablet_bootstrap.cc:492] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b: Bootstrap starting.
I20260621 02:45:06.916428 18066 tablet_bootstrap.cc:654] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:06.919589 18066 tablet_bootstrap.cc:492] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b: No bootstrap required, opened a new log
I20260621 02:45:06.919749 18066 ts_tablet_manager.cc:1397] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260621 02:45:06.920853 18066 raft_consensus.cc:348] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.921195 18066 raft_consensus.cc:374] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:06.921325 18066 raft_consensus.cc:729] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b20a1a36b827478fb586a00ec935b92b, State: Initialized, Role: FOLLOWER
I20260621 02:45:06.921685 18066 consensus_queue.cc:260] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.922477 18066 ts_tablet_manager.cc:1428] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260621 02:45:06.923336 18065 tablet_bootstrap.cc:492] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c: Bootstrap starting.
I20260621 02:45:06.925047 18065 tablet_bootstrap.cc:654] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:06.929150 18065 tablet_bootstrap.cc:492] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c: No bootstrap required, opened a new log
I20260621 02:45:06.929319 18065 ts_tablet_manager.cc:1397] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260621 02:45:06.929916 18065 raft_consensus.cc:348] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.930136 18065 raft_consensus.cc:374] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:06.930254 18065 raft_consensus.cc:729] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf48897dfda844d685b5b2540558361c, State: Initialized, Role: FOLLOWER
I20260621 02:45:06.930517 18065 consensus_queue.cc:260] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.931020 18065 ts_tablet_manager.cc:1428] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260621 02:45:06.940877 18007 raft_consensus.cc:482] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:45:06.941061 18007 raft_consensus.cc:504] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.941814 18007 leader_election.cc:288] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8460ee014fce4c72935c507fb0a96606 (127.9.166.1:32967), b20a1a36b827478fb586a00ec935b92b (127.9.166.3:43225)
I20260621 02:45:06.942154 18042 raft_consensus.cc:482] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:45:06.942363 18042 raft_consensus.cc:504] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.942447 17640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3ffcf3209a417d825a5b17a800a4e9" candidate_uuid: "cf48897dfda844d685b5b2540558361c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8460ee014fce4c72935c507fb0a96606" is_pre_election: true
I20260621 02:45:06.942576 17918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3ffcf3209a417d825a5b17a800a4e9" candidate_uuid: "cf48897dfda844d685b5b2540558361c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b20a1a36b827478fb586a00ec935b92b" is_pre_election: true
I20260621 02:45:06.942687 17640 raft_consensus.cc:2454] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf48897dfda844d685b5b2540558361c in term 0.
I20260621 02:45:06.943032 17918 raft_consensus.cc:2454] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf48897dfda844d685b5b2540558361c in term 0.
I20260621 02:45:06.943142 17706 leader_election.cc:302] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8460ee014fce4c72935c507fb0a96606, cf48897dfda844d685b5b2540558361c; no voters:
I20260621 02:45:06.943457 18007 raft_consensus.cc:2781] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:45:06.943571 18007 raft_consensus.cc:482] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:45:06.943713 18007 raft_consensus.cc:3037] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:06.944028 18042 leader_election.cc:288] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8460ee014fce4c72935c507fb0a96606 (127.9.166.1:32967), cf48897dfda844d685b5b2540558361c (127.9.166.2:37627)
I20260621 02:45:06.944109 17640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3ffcf3209a417d825a5b17a800a4e9" candidate_uuid: "b20a1a36b827478fb586a00ec935b92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8460ee014fce4c72935c507fb0a96606" is_pre_election: true
I20260621 02:45:06.944432 17640 raft_consensus.cc:2454] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b20a1a36b827478fb586a00ec935b92b in term 0.
I20260621 02:45:06.944510 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3ffcf3209a417d825a5b17a800a4e9" candidate_uuid: "b20a1a36b827478fb586a00ec935b92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf48897dfda844d685b5b2540558361c" is_pre_election: true
I20260621 02:45:06.945247 17845 leader_election.cc:302] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8460ee014fce4c72935c507fb0a96606, b20a1a36b827478fb586a00ec935b92b; no voters:
I20260621 02:45:06.946241 18042 raft_consensus.cc:2781] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:45:06.946377 18042 raft_consensus.cc:482] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:45:06.946498 18042 raft_consensus.cc:3037] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:06.947163 18007 raft_consensus.cc:504] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.947606 17775 raft_consensus.cc:2379] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b20a1a36b827478fb586a00ec935b92b in current term 1: Already voted for candidate cf48897dfda844d685b5b2540558361c in this term.
I20260621 02:45:06.947785 18007 leader_election.cc:288] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [CANDIDATE]: Term 1 election: Requested vote from peers 8460ee014fce4c72935c507fb0a96606 (127.9.166.1:32967), b20a1a36b827478fb586a00ec935b92b (127.9.166.3:43225)
I20260621 02:45:06.948494 17640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3ffcf3209a417d825a5b17a800a4e9" candidate_uuid: "cf48897dfda844d685b5b2540558361c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8460ee014fce4c72935c507fb0a96606"
I20260621 02:45:06.948684 17640 raft_consensus.cc:3037] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:06.948978 17918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3ffcf3209a417d825a5b17a800a4e9" candidate_uuid: "cf48897dfda844d685b5b2540558361c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b20a1a36b827478fb586a00ec935b92b"
I20260621 02:45:06.949520 18042 raft_consensus.cc:504] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.949942 17918 raft_consensus.cc:2379] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cf48897dfda844d685b5b2540558361c in current term 1: Already voted for candidate b20a1a36b827478fb586a00ec935b92b in this term.
I20260621 02:45:06.950505 17639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3ffcf3209a417d825a5b17a800a4e9" candidate_uuid: "b20a1a36b827478fb586a00ec935b92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8460ee014fce4c72935c507fb0a96606"
I20260621 02:45:06.951438 18042 leader_election.cc:288] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b [CANDIDATE]: Term 1 election: Requested vote from peers 8460ee014fce4c72935c507fb0a96606 (127.9.166.1:32967), cf48897dfda844d685b5b2540558361c (127.9.166.2:37627)
I20260621 02:45:06.951730 17640 raft_consensus.cc:2454] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf48897dfda844d685b5b2540558361c in term 1.
I20260621 02:45:06.952081 17706 leader_election.cc:302] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8460ee014fce4c72935c507fb0a96606, cf48897dfda844d685b5b2540558361c; no voters: b20a1a36b827478fb586a00ec935b92b
I20260621 02:45:06.952139 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3ffcf3209a417d825a5b17a800a4e9" candidate_uuid: "b20a1a36b827478fb586a00ec935b92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf48897dfda844d685b5b2540558361c"
I20260621 02:45:06.952387 17775 raft_consensus.cc:2379] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b20a1a36b827478fb586a00ec935b92b in current term 1: Already voted for candidate cf48897dfda844d685b5b2540558361c in this term.
I20260621 02:45:06.952512 18007 raft_consensus.cc:2781] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:45:06.952905 17845 leader_election.cc:302] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b20a1a36b827478fb586a00ec935b92b; no voters: 8460ee014fce4c72935c507fb0a96606, cf48897dfda844d685b5b2540558361c
I20260621 02:45:06.953145 18042 raft_consensus.cc:2726] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260621 02:45:06.954015 18007 raft_consensus.cc:686] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [term 1 LEADER]: Becoming Leader. State: Replica: cf48897dfda844d685b5b2540558361c, State: Running, Role: LEADER
I20260621 02:45:06.954291 18007 consensus_queue.cc:237] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:06.956882 17479 catalog_manager.cc:5495] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c reported cstate change: term changed from 0 to 1, leader changed from <none> to cf48897dfda844d685b5b2540558361c (127.9.166.2). New cstate: current_term: 1 leader_uuid: "cf48897dfda844d685b5b2540558361c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } health_report { overall_health: UNKNOWN } } }
I20260621 02:45:06.973438 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:06.966730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.9.166.2:37627)
0621 02:45:06.967362 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:06.967368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:06.967417 (+ 49us) server_negotiation.cc:407] Connection header received
0621 02:45:06.967482 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:06.967487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:06.967570 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:06.967668 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:45:06.968094 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.969256 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.971176 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:06.971916 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:06.972052 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:06.972588 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:45:06.972699 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:45:06.972803 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:06.972989 (+ 186us) server_negotiation.cc:299] Negotiation successful
0621 02:45:06.973067 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":363,"threads_started":1}
I20260621 02:45:06.976346 17640 raft_consensus.cc:1261] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606 [term 1 FOLLOWER]: Refusing update from remote peer cf48897dfda844d685b5b2540558361c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:45:06.976426 17918 raft_consensus.cc:1261] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b [term 1 FOLLOWER]: Refusing update from remote peer cf48897dfda844d685b5b2540558361c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:45:06.977295 18069 consensus_queue.cc:1035] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:45:06.977725 18007 consensus_queue.cc:1035] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:45:06.997234 17478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:45:06.997526 17478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:45:07.000638 17478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5: Sending DeleteTablet for 3 replicas of tablet 8a3ffcf3209a417d825a5b17a800a4e9
I20260621 02:45:07.001492 17616 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a3ffcf3209a417d825a5b17a800a4e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:06 UTC) from {username='oryx', principal='oryx/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:35826
I20260621 02:45:07.001585 17893 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a3ffcf3209a417d825a5b17a800a4e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:06 UTC) from {username='oryx', principal='oryx/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:51578
I20260621 02:45:07.001827 18049 tablet_replica.cc:330] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606: stopping tablet replica
I20260621 02:45:07.001878 18048 tablet_replica.cc:330] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b: stopping tablet replica
I20260621 02:45:07.002071 18049 raft_consensus.cc:2229] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:45:07.002099 18048 raft_consensus.cc:2229] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:45:07.002314 18049 raft_consensus.cc:2258] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:07.002364 18048 raft_consensus.cc:2258] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:07.002813 17755 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a3ffcf3209a417d825a5b17a800a4e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:06 UTC) from {username='oryx', principal='oryx/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:51850
I20260621 02:45:07.003208 18050 tablet_replica.cc:330] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c: stopping tablet replica
I20260621 02:45:07.003458 18050 raft_consensus.cc:2229] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:45:07.003710 18049 ts_tablet_manager.cc:1905] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:45:07.003784 18050 raft_consensus.cc:2258] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:07.004305 17478 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:45:07.004865 17478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:45:07.007033 18048 ts_tablet_manager.cc:1905] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:45:07.010568 18050 ts_tablet_manager.cc:1905] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:45:07.010785 18049 ts_tablet_manager.cc:1918] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:45:07.010924 18049 log.cc:1196] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/8a3ffcf3209a417d825a5b17a800a4e9
I20260621 02:45:07.011333 18049 ts_tablet_manager.cc:1939] T 8a3ffcf3209a417d825a5b17a800a4e9 P 8460ee014fce4c72935c507fb0a96606: Deleting consensus metadata
I20260621 02:45:07.012426 17463 catalog_manager.cc:4841] TS 8460ee014fce4c72935c507fb0a96606 (127.9.166.1:32967): tablet 8a3ffcf3209a417d825a5b17a800a4e9 (table test-table [id=83cb0881353d42b0a35151bb3539fc8c]) successfully deleted
I20260621 02:45:07.015842 18050 ts_tablet_manager.cc:1918] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:45:07.016000 18050 log.cc:1196] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/8a3ffcf3209a417d825a5b17a800a4e9
I20260621 02:45:07.016487 18050 ts_tablet_manager.cc:1939] T 8a3ffcf3209a417d825a5b17a800a4e9 P cf48897dfda844d685b5b2540558361c: Deleting consensus metadata
I20260621 02:45:07.017486 17463 catalog_manager.cc:4841] TS cf48897dfda844d685b5b2540558361c (127.9.166.2:37627): tablet 8a3ffcf3209a417d825a5b17a800a4e9 (table test-table [id=83cb0881353d42b0a35151bb3539fc8c]) successfully deleted
I20260621 02:45:07.018400 17616 tablet_service.cc:1467] Processing CreateTablet for tablet ba4f07c12e2a4f7f897415363dc93b86 (DEFAULT_TABLE table=test-table [id=686b2f1df03447febed4a8deba8d2ad2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:45:07.018833 17616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba4f07c12e2a4f7f897415363dc93b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:07.018909 17893 tablet_service.cc:1467] Processing CreateTablet for tablet ba4f07c12e2a4f7f897415363dc93b86 (DEFAULT_TABLE table=test-table [id=686b2f1df03447febed4a8deba8d2ad2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:45:07.019312 17893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba4f07c12e2a4f7f897415363dc93b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:07.020869 18048 ts_tablet_manager.cc:1918] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:45:07.021037 18048 log.cc:1196] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/8a3ffcf3209a417d825a5b17a800a4e9
I20260621 02:45:07.021544 18048 ts_tablet_manager.cc:1939] T 8a3ffcf3209a417d825a5b17a800a4e9 P b20a1a36b827478fb586a00ec935b92b: Deleting consensus metadata
I20260621 02:45:07.022722 17465 catalog_manager.cc:4841] TS b20a1a36b827478fb586a00ec935b92b (127.9.166.3:43225): tablet 8a3ffcf3209a417d825a5b17a800a4e9 (table test-table [id=83cb0881353d42b0a35151bb3539fc8c]) successfully deleted
I20260621 02:45:07.025079 18063 tablet_bootstrap.cc:492] T ba4f07c12e2a4f7f897415363dc93b86 P 8460ee014fce4c72935c507fb0a96606: Bootstrap starting.
I20260621 02:45:07.025915 17755 tablet_service.cc:1467] Processing CreateTablet for tablet ba4f07c12e2a4f7f897415363dc93b86 (DEFAULT_TABLE table=test-table [id=686b2f1df03447febed4a8deba8d2ad2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:45:07.026309 17755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba4f07c12e2a4f7f897415363dc93b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:07.026821 18063 tablet_bootstrap.cc:654] T ba4f07c12e2a4f7f897415363dc93b86 P 8460ee014fce4c72935c507fb0a96606: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:07.028874 18063 tablet_bootstrap.cc:492] T ba4f07c12e2a4f7f897415363dc93b86 P 8460ee014fce4c72935c507fb0a96606: No bootstrap required, opened a new log
I20260621 02:45:07.029042 18063 ts_tablet_manager.cc:1397] T ba4f07c12e2a4f7f897415363dc93b86 P 8460ee014fce4c72935c507fb0a96606: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260621 02:45:07.029809 18063 raft_consensus.cc:348] T ba4f07c12e2a4f7f897415363dc93b86 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:07.030077 18063 raft_consensus.cc:374] T ba4f07c12e2a4f7f897415363dc93b86 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:07.030167 18063 raft_consensus.cc:729] T ba4f07c12e2a4f7f897415363dc93b86 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8460ee014fce4c72935c507fb0a96606, State: Initialized, Role: FOLLOWER
I20260621 02:45:07.030443 18063 consensus_queue.cc:260] T ba4f07c12e2a4f7f897415363dc93b86 P 8460ee014fce4c72935c507fb0a96606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:07.031015 18063 ts_tablet_manager.cc:1428] T ba4f07c12e2a4f7f897415363dc93b86 P 8460ee014fce4c72935c507fb0a96606: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:45:07.031834 18065 tablet_bootstrap.cc:492] T ba4f07c12e2a4f7f897415363dc93b86 P cf48897dfda844d685b5b2540558361c: Bootstrap starting.
I20260621 02:45:07.033078 18065 tablet_bootstrap.cc:654] T ba4f07c12e2a4f7f897415363dc93b86 P cf48897dfda844d685b5b2540558361c: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:07.035472 18065 tablet_bootstrap.cc:492] T ba4f07c12e2a4f7f897415363dc93b86 P cf48897dfda844d685b5b2540558361c: No bootstrap required, opened a new log
I20260621 02:45:07.035638 18065 ts_tablet_manager.cc:1397] T ba4f07c12e2a4f7f897415363dc93b86 P cf48897dfda844d685b5b2540558361c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260621 02:45:07.036199 18065 raft_consensus.cc:348] T ba4f07c12e2a4f7f897415363dc93b86 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:07.036396 18065 raft_consensus.cc:374] T ba4f07c12e2a4f7f897415363dc93b86 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:07.036495 18065 raft_consensus.cc:729] T ba4f07c12e2a4f7f897415363dc93b86 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf48897dfda844d685b5b2540558361c, State: Initialized, Role: FOLLOWER
I20260621 02:45:07.036748 18065 consensus_queue.cc:260] T ba4f07c12e2a4f7f897415363dc93b86 P cf48897dfda844d685b5b2540558361c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:07.036943 18066 tablet_bootstrap.cc:492] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b: Bootstrap starting.
I20260621 02:45:07.038740 18066 tablet_bootstrap.cc:654] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:07.039283 18065 ts_tablet_manager.cc:1428] T ba4f07c12e2a4f7f897415363dc93b86 P cf48897dfda844d685b5b2540558361c: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260621 02:45:07.040823 18066 tablet_bootstrap.cc:492] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b: No bootstrap required, opened a new log
I20260621 02:45:07.040982 18066 ts_tablet_manager.cc:1397] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260621 02:45:07.041790 18066 raft_consensus.cc:348] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:07.042058 18066 raft_consensus.cc:374] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:07.042140 18066 raft_consensus.cc:729] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b20a1a36b827478fb586a00ec935b92b, State: Initialized, Role: FOLLOWER
I20260621 02:45:07.042421 18066 consensus_queue.cc:260] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:07.043036 18066 ts_tablet_manager.cc:1428] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260621 02:45:07.165958 18042 raft_consensus.cc:482] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:45:07.166178 18042 raft_consensus.cc:504] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:07.166846 18042 leader_election.cc:288] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8460ee014fce4c72935c507fb0a96606 (127.9.166.1:32967), cf48897dfda844d685b5b2540558361c (127.9.166.2:37627)
I20260621 02:45:07.167526 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba4f07c12e2a4f7f897415363dc93b86" candidate_uuid: "b20a1a36b827478fb586a00ec935b92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf48897dfda844d685b5b2540558361c" is_pre_election: true
I20260621 02:45:07.167533 17640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba4f07c12e2a4f7f897415363dc93b86" candidate_uuid: "b20a1a36b827478fb586a00ec935b92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8460ee014fce4c72935c507fb0a96606" is_pre_election: true
I20260621 02:45:07.167780 17775 raft_consensus.cc:2454] T ba4f07c12e2a4f7f897415363dc93b86 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b20a1a36b827478fb586a00ec935b92b in term 0.
I20260621 02:45:07.167786 17640 raft_consensus.cc:2454] T ba4f07c12e2a4f7f897415363dc93b86 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b20a1a36b827478fb586a00ec935b92b in term 0.
I20260621 02:45:07.168220 17845 leader_election.cc:302] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b20a1a36b827478fb586a00ec935b92b, cf48897dfda844d685b5b2540558361c; no voters:
I20260621 02:45:07.168493 18042 raft_consensus.cc:2781] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:45:07.168627 18042 raft_consensus.cc:482] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:45:07.168754 18042 raft_consensus.cc:3037] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:07.170873 18042 raft_consensus.cc:504] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:07.171446 18042 leader_election.cc:288] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [CANDIDATE]: Term 1 election: Requested vote from peers 8460ee014fce4c72935c507fb0a96606 (127.9.166.1:32967), cf48897dfda844d685b5b2540558361c (127.9.166.2:37627)
I20260621 02:45:07.171957 17640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba4f07c12e2a4f7f897415363dc93b86" candidate_uuid: "b20a1a36b827478fb586a00ec935b92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8460ee014fce4c72935c507fb0a96606"
I20260621 02:45:07.172019 17775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba4f07c12e2a4f7f897415363dc93b86" candidate_uuid: "b20a1a36b827478fb586a00ec935b92b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf48897dfda844d685b5b2540558361c"
I20260621 02:45:07.172197 17775 raft_consensus.cc:3037] T ba4f07c12e2a4f7f897415363dc93b86 P cf48897dfda844d685b5b2540558361c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:07.172189 17640 raft_consensus.cc:3037] T ba4f07c12e2a4f7f897415363dc93b86 P 8460ee014fce4c72935c507fb0a96606 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:07.174144 17640 raft_consensus.cc:2454] T ba4f07c12e2a4f7f897415363dc93b86 P 8460ee014fce4c72935c507fb0a96606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b20a1a36b827478fb586a00ec935b92b in term 1.
I20260621 02:45:07.174525 17845 leader_election.cc:302] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8460ee014fce4c72935c507fb0a96606, b20a1a36b827478fb586a00ec935b92b; no voters:
I20260621 02:45:07.174818 18042 raft_consensus.cc:2781] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:45:07.174978 18042 raft_consensus.cc:686] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [term 1 LEADER]: Becoming Leader. State: Replica: b20a1a36b827478fb586a00ec935b92b, State: Running, Role: LEADER
I20260621 02:45:07.175235 18042 consensus_queue.cc:237] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } }
I20260621 02:45:07.175414 17775 raft_consensus.cc:2454] T ba4f07c12e2a4f7f897415363dc93b86 P cf48897dfda844d685b5b2540558361c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b20a1a36b827478fb586a00ec935b92b in term 1.
I20260621 02:45:07.178092 17479 catalog_manager.cc:5495] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b reported cstate change: term changed from 0 to 1, leader changed from <none> to b20a1a36b827478fb586a00ec935b92b (127.9.166.3). New cstate: current_term: 1 leader_uuid: "b20a1a36b827478fb586a00ec935b92b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b20a1a36b827478fb586a00ec935b92b" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43225 } health_report { overall_health: HEALTHY } } }
I20260621 02:45:07.197568 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:07.192028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.9.166.3:43225)
0621 02:45:07.192199 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:07.192206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:07.192308 (+ 102us) server_negotiation.cc:407] Connection header received
0621 02:45:07.192448 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:07.192453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:07.192532 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:07.192642 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:45:07.192919 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:07.194081 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:07.195570 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:07.196308 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:07.196441 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:07.196852 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:45:07.196980 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:45:07.197068 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:07.197175 (+ 107us) server_negotiation.cc:299] Negotiation successful
0621 02:45:07.197248 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 02:45:07.206825 17775 raft_consensus.cc:1261] T ba4f07c12e2a4f7f897415363dc93b86 P cf48897dfda844d685b5b2540558361c [term 1 FOLLOWER]: Refusing update from remote peer b20a1a36b827478fb586a00ec935b92b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:45:07.206947 17640 raft_consensus.cc:1261] T ba4f07c12e2a4f7f897415363dc93b86 P 8460ee014fce4c72935c507fb0a96606 [term 1 FOLLOWER]: Refusing update from remote peer b20a1a36b827478fb586a00ec935b92b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:45:07.207373 18045 consensus_queue.cc:1035] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf48897dfda844d685b5b2540558361c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:45:07.207739 18019 consensus_queue.cc:1035] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8460ee014fce4c72935c507fb0a96606" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 32967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:45:07.258123 17478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:45:07.258365 17478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:45:07.261365 17478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7bb31d65aae4bfd8d9ff296a1a9cda5: Sending DeleteTablet for 3 replicas of tablet ba4f07c12e2a4f7f897415363dc93b86
I20260621 02:45:07.262230 17616 tablet_service.cc:1514] Processing DeleteTablet for tablet ba4f07c12e2a4f7f897415363dc93b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:07 UTC) from {username='oryx', principal='oryx/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:35826
I20260621 02:45:07.262472 17755 tablet_service.cc:1514] Processing DeleteTablet for tablet ba4f07c12e2a4f7f897415363dc93b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:07 UTC) from {username='oryx', principal='oryx/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:51850
I20260621 02:45:07.262784 18050 tablet_replica.cc:330] T ba4f07c12e2a4f7f897415363dc93b86 P cf48897dfda844d685b5b2540558361c: stopping tablet replica
I20260621 02:45:07.262956 18050 raft_consensus.cc:2229] T ba4f07c12e2a4f7f897415363dc93b86 P cf48897dfda844d685b5b2540558361c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:45:07.263173 18050 raft_consensus.cc:2258] T ba4f07c12e2a4f7f897415363dc93b86 P cf48897dfda844d685b5b2540558361c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:07.263206 18049 tablet_replica.cc:330] T ba4f07c12e2a4f7f897415363dc93b86 P 8460ee014fce4c72935c507fb0a96606: stopping tablet replica
I20260621 02:45:07.263374 18049 raft_consensus.cc:2229] T ba4f07c12e2a4f7f897415363dc93b86 P 8460ee014fce4c72935c507fb0a96606 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:45:07.263687 18049 raft_consensus.cc:2258] T ba4f07c12e2a4f7f897415363dc93b86 P 8460ee014fce4c72935c507fb0a96606 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:07.265199 18049 ts_tablet_manager.cc:1905] T ba4f07c12e2a4f7f897415363dc93b86 P 8460ee014fce4c72935c507fb0a96606: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:45:07.272324 18050 ts_tablet_manager.cc:1905] T ba4f07c12e2a4f7f897415363dc93b86 P cf48897dfda844d685b5b2540558361c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:45:07.271116 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 17550
I20260621 02:45:07.278849 18050 ts_tablet_manager.cc:1918] T ba4f07c12e2a4f7f897415363dc93b86 P cf48897dfda844d685b5b2540558361c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:45:07.278973 18050 log.cc:1196] T ba4f07c12e2a4f7f897415363dc93b86 P cf48897dfda844d685b5b2540558361c: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/ba4f07c12e2a4f7f897415363dc93b86
I20260621 02:45:07.279349 18050 ts_tablet_manager.cc:1939] T ba4f07c12e2a4f7f897415363dc93b86 P cf48897dfda844d685b5b2540558361c: Deleting consensus metadata
I20260621 02:45:07.280380 17463 catalog_manager.cc:4841] TS cf48897dfda844d685b5b2540558361c (127.9.166.2:37627): tablet ba4f07c12e2a4f7f897415363dc93b86 (table test-table [id=686b2f1df03447febed4a8deba8d2ad2]) successfully deleted
I20260621 02:45:07.280977 17893 tablet_service.cc:1514] Processing DeleteTablet for tablet ba4f07c12e2a4f7f897415363dc93b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:07 UTC) from {username='oryx', principal='oryx/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:51578
I20260621 02:45:07.281566 18049 ts_tablet_manager.cc:1918] T ba4f07c12e2a4f7f897415363dc93b86 P 8460ee014fce4c72935c507fb0a96606: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:45:07.281695 18049 log.cc:1196] T ba4f07c12e2a4f7f897415363dc93b86 P 8460ee014fce4c72935c507fb0a96606: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/ba4f07c12e2a4f7f897415363dc93b86
I20260621 02:45:07.282107 18049 ts_tablet_manager.cc:1939] T ba4f07c12e2a4f7f897415363dc93b86 P 8460ee014fce4c72935c507fb0a96606: Deleting consensus metadata
I20260621 02:45:07.283111 17463 catalog_manager.cc:4841] TS 8460ee014fce4c72935c507fb0a96606 (127.9.166.1:32967): tablet ba4f07c12e2a4f7f897415363dc93b86 (table test-table [id=686b2f1df03447febed4a8deba8d2ad2]) successfully deleted
I20260621 02:45:07.283445 18048 tablet_replica.cc:330] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b: stopping tablet replica
I20260621 02:45:07.283636 18048 raft_consensus.cc:2229] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:45:07.284042 18048 raft_consensus.cc:2258] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:07.285266 18048 ts_tablet_manager.cc:1905] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:45:07.292178 18048 ts_tablet_manager.cc:1918] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260621 02:45:07.292320 18048 log.cc:1196] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/ba4f07c12e2a4f7f897415363dc93b86
I20260621 02:45:07.292737 18048 ts_tablet_manager.cc:1939] T ba4f07c12e2a4f7f897415363dc93b86 P b20a1a36b827478fb586a00ec935b92b: Deleting consensus metadata
I20260621 02:45:07.293809 17465 catalog_manager.cc:4841] TS b20a1a36b827478fb586a00ec935b92b (127.9.166.3:43225): tablet ba4f07c12e2a4f7f897415363dc93b86 (table test-table [id=686b2f1df03447febed4a8deba8d2ad2]) successfully deleted
Jun 21 02:45:07 dist-test-slave-hvgn krb5kdc[17434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.166.1@KRBTEST.COM
I20260621 02:45:07.314399 18081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:07.275592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35830 (local address 127.9.166.1:32967)
0621 02:45:07.299814 (+ 24222us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:07.299818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:07.299834 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:45:07.299882 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:07.299886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:07.299943 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:07.300066 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:07.301402 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:07.302169 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:07.303572 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:07.304109 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:07.304235 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:07.308143 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:07.308156 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:07.308172 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:07.308207 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:07.311322 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:07.312112 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:07.312128 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:07.312130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:07.312268 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:07.312704 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:07.312709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:07.312718 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:07.313113 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:07.313369 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:07.313832 (+ 463us) server_negotiation.cc:299] Negotiation successful
0621 02:45:07.314047 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24115,"thread_start_us":24000,"threads_started":1}
I20260621 02:45:07.315053 17681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:07.433547 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 17550
I20260621 02:45:07.452327 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 17690
Jun 21 02:45:07 dist-test-slave-hvgn krb5kdc[17434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.166.2@KRBTEST.COM
I20260621 02:45:07.467778 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:07.453140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.9.166.2:37627)
0621 02:45:07.453308 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:07.453315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:07.453362 (+ 47us) server_negotiation.cc:407] Connection header received
0621 02:45:07.453592 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:07.453598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:07.453673 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:07.453788 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:07.454973 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:07.455783 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:07.456981 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:07.457391 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:07.457535 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:07.461942 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:07.461955 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:07.461971 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:07.462004 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:07.464927 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:07.465610 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:07.465619 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:07.465621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:07.465753 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:07.466126 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:07.466132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:07.466136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:07.466486 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:07.466811 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:07.467079 (+ 268us) server_negotiation.cc:299] Negotiation successful
0621 02:45:07.467302 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:45:07.468426 17820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:07.590867 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 17690
I20260621 02:45:07.613605 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 17829
Jun 21 02:45:07 dist-test-slave-hvgn krb5kdc[17434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.166.3@KRBTEST.COM
I20260621 02:45:07.627950 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:07.614291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51616 (local address 127.9.166.3:43225)
0621 02:45:07.614491 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:07.614495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:07.614513 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:45:07.614665 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:07.614669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:07.614738 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:07.614820 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:07.616092 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:07.616804 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:07.617955 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:07.618389 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:07.618486 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:07.622645 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:07.622664 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:07.622679 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:07.622709 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:07.625370 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:07.625952 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:07.625959 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:07.625961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:07.626129 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:07.626466 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:07.626470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:07.626472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:07.626824 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:07.627052 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:07.627283 (+ 231us) server_negotiation.cc:299] Negotiation successful
0621 02:45:07.627625 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 02:45:07.628587 17959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:07.790114 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 17829
I20260621 02:45:07.815271 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 17447
I20260621 02:45:07.816494 17519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:07.931437 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 17447
2026-06-21T02:45:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5385 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:45:08 dist-test-slave-hvgn krb5kdc[18090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:45:08 dist-test-slave-hvgn krb5kdc[18090](info): set up 2 sockets
Jun 21 02:45:08 dist-test-slave-hvgn krb5kdc[18090](info): commencing operation
krb5kdc: starting...
W20260621 02:45:09.024691 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:45:09 dist-test-slave-hvgn krb5kdc[18090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:45:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:45:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:45:09.231006 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:39549
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:38975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:39549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:45:09.547317 18103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:09.547698 18103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:09.547860 18103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:09.555111 18103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:45:09.555265 18103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:09.555375 18103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:09.555505 18103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:45:09.555586 18103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:45:09.565593 18103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:38975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:39549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:39549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:09.566941 18103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:09.568398 18103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:09.580736 18108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:09.580696 18109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:09.580982 18111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:09.581637 18103 server_base.cc:1034] running on GCE node
I20260621 02:45:09.582396 18103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:09.583909 18103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:09.585151 18103 hybrid_clock.cc:648] HybridClock initialized: now 1782009909585101 us; error 64 us; skew 500 ppm
I20260621 02:45:09.585502 18103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-21T02:45:09Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1723 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:45:09 dist-test-slave-hvgn krb5kdc[18117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:45:09 dist-test-slave-hvgn krb5kdc[18117](info): set up 2 sockets
Jun 21 02:45:09 dist-test-slave-hvgn krb5kdc[18117](info): commencing operation
krb5kdc: starting...
W20260621 02:45:10.748256 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:45:10 dist-test-slave-hvgn krb5kdc[18117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:45:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:45:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:45:10.959272 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:34841
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:34841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:45:11.273597 18130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:11.273872 18130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:11.273965 18130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:11.281152 18130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:45:11.281250 18130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:11.281296 18130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:11.281350 18130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:45:11.281387 18130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:45:11.291061 18130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:34841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:34841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:11.292379 18130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:11.294100 18130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:11.307536 18136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:11.307559 18135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:11.307495 18138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:11.308350 18130 server_base.cc:1034] running on GCE node
I20260621 02:45:11.309108 18130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:11.310460 18130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:11.311677 18130 hybrid_clock.cc:648] HybridClock initialized: now 1782009911311643 us; error 46 us; skew 500 ppm
I20260621 02:45:11.312012 18130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:45:11 dist-test-slave-hvgn krb5kdc[18117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009911, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:45:11.314975 18130 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:45:11.316473 18130 webserver.cc:460] Webserver started at http://127.9.166.62:37705/ using document root <none> and password file <none>
I20260621 02:45:11.317147 18130 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:11.317286 18130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:11.317598 18130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:45:11.320514 18130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "a31555c7897d46ae92da555498d0a070"
format_stamp: "Formatted at 2026-06-21 02:45:11 on dist-test-slave-hvgn"
server_key: "d49ca6cd7c91e22bb304e471304f0b2c"
server_key_iv: "0d284dbba153a9de26300e046fce71b2"
server_key_version: "encryptionkey@0"
I20260621 02:45:11.321259 18130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "a31555c7897d46ae92da555498d0a070"
format_stamp: "Formatted at 2026-06-21 02:45:11 on dist-test-slave-hvgn"
server_key: "d49ca6cd7c91e22bb304e471304f0b2c"
server_key_iv: "0d284dbba153a9de26300e046fce71b2"
server_key_version: "encryptionkey@0"
I20260621 02:45:11.326216 18130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260621 02:45:11.329679 18145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:11.330966 18130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 02:45:11.331147 18130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "a31555c7897d46ae92da555498d0a070"
format_stamp: "Formatted at 2026-06-21 02:45:11 on dist-test-slave-hvgn"
server_key: "d49ca6cd7c91e22bb304e471304f0b2c"
server_key_iv: "0d284dbba153a9de26300e046fce71b2"
server_key_version: "encryptionkey@0"
I20260621 02:45:11.331339 18130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:11.365513 18130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:11.369674 18130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:11.369967 18130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:11.388646 18130 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:34841
I20260621 02:45:11.388693 18196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:34841 every 8 connection(s)
I20260621 02:45:11.389989 18130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:45:11.393635 18197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:11.399036 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 18130
I20260621 02:45:11.399240 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:45:11.400069 9880 external_mini_cluster.cc:1428] Setting key feb68ce756bbc801992ece5b1a652106
I20260621 02:45:11.405004 18197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070: Bootstrap starting.
I20260621 02:45:11.407990 18197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:11.408982 18197 log.cc:824] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070: Log is configured to *not* fsync() on all Append() calls
I20260621 02:45:11.411852 18197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070: No bootstrap required, opened a new log
Jun 21 02:45:11 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:45:11.420970 18197 raft_consensus.cc:348] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31555c7897d46ae92da555498d0a070" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34841 } }
I20260621 02:45:11.421393 18197 raft_consensus.cc:374] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:11.421521 18197 raft_consensus.cc:729] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a31555c7897d46ae92da555498d0a070, State: Initialized, Role: FOLLOWER
I20260621 02:45:11.422125 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:11.402155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38964 (local address 127.9.166.62:34841)
0621 02:45:11.402952 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:11.402967 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:11.403019 (+ 52us) server_negotiation.cc:407] Connection header received
0621 02:45:11.404671 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:11.404694 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:11.405106 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:11.405507 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:11.406911 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:11.408289 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:11.409650 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:11.410502 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:11.410683 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:11.414974 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:11.415013 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:11.415031 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:11.415081 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:11.418150 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:11.418922 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:11.418928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:11.418933 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:11.419113 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:11.419520 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:11.419523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:11.419525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:11.420019 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:11.420213 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:11.420655 (+ 442us) server_negotiation.cc:299] Negotiation successful
0621 02:45:11.420948 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":366,"threads_started":1}
I20260621 02:45:11.422094 18197 consensus_queue.cc:260] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31555c7897d46ae92da555498d0a070" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34841 } }
I20260621 02:45:11.422354 18197 raft_consensus.cc:388] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:45:11.422566 18197 raft_consensus.cc:482] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:45:11.422752 18197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:11.425124 18197 raft_consensus.cc:504] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31555c7897d46ae92da555498d0a070" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34841 } }
I20260621 02:45:11.425573 18197 leader_election.cc:302] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a31555c7897d46ae92da555498d0a070; no voters:
I20260621 02:45:11.426091 18197 leader_election.cc:288] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:45:11.426222 18202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:45:11.426892 18202 raft_consensus.cc:686] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [term 1 LEADER]: Becoming Leader. State: Replica: a31555c7897d46ae92da555498d0a070, State: Running, Role: LEADER
I20260621 02:45:11.427455 18202 consensus_queue.cc:237] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31555c7897d46ae92da555498d0a070" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34841 } }
I20260621 02:45:11.428181 18197 sys_catalog.cc:564] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:45:11.429924 18203 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a31555c7897d46ae92da555498d0a070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31555c7897d46ae92da555498d0a070" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34841 } } }
I20260621 02:45:11.430290 18203 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [sys.catalog]: This master's current role is: LEADER
I20260621 02:45:11.430523 18204 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a31555c7897d46ae92da555498d0a070. Latest consensus state: current_term: 1 leader_uuid: "a31555c7897d46ae92da555498d0a070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31555c7897d46ae92da555498d0a070" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34841 } } }
I20260621 02:45:11.431005 18204 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [sys.catalog]: This master's current role is: LEADER
I20260621 02:45:11.435042 18207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:45:11.440372 18207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:45:11.449169 18207 catalog_manager.cc:1347] Generated new cluster ID: 8dc39c6891f9487f8785298ad98412ae
I20260621 02:45:11.449275 18207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:45:11.457651 18207 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:45:11.458940 18207 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:45:11.464903 18207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070: Generated new TSK 0
I20260621 02:45:11.465719 18207 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
I20260621 02:45:11.515326 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34841
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:45:11.836500 18223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:11.836807 18223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:11.836949 18223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:11.844220 18223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:11.844334 18223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:11.844496 18223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:45:11.854383 18223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:11.855741 18223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:11.857069 18223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:11.869982 18231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:11.870023 18229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:11.870064 18228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:11.872758 18223 server_base.cc:1034] running on GCE node
I20260621 02:45:11.873476 18223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:11.874517 18223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:11.875739 18223 hybrid_clock.cc:648] HybridClock initialized: now 1782009911875688 us; error 49 us; skew 500 ppm
I20260621 02:45:11.876046 18223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:45:11 dist-test-slave-hvgn krb5kdc[18117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009911, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:45:11.883654 18223 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:45:11.885181 18223 webserver.cc:460] Webserver started at http://127.9.166.1:46033/ using document root <none> and password file <none>
I20260621 02:45:11.885888 18223 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:11.886032 18223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:11.886338 18223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:45:11.889344 18223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "7f0d30dbbafd495caadc58dc4979f949"
format_stamp: "Formatted at 2026-06-21 02:45:11 on dist-test-slave-hvgn"
server_key: "f96c2f7edf41caf630613d4bab9b5d29"
server_key_iv: "0709dc65f0829843ca65b277fb4a62eb"
server_key_version: "encryptionkey@0"
I20260621 02:45:11.890096 18223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "7f0d30dbbafd495caadc58dc4979f949"
format_stamp: "Formatted at 2026-06-21 02:45:11 on dist-test-slave-hvgn"
server_key: "f96c2f7edf41caf630613d4bab9b5d29"
server_key_iv: "0709dc65f0829843ca65b277fb4a62eb"
server_key_version: "encryptionkey@0"
I20260621 02:45:11.894990 18223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:45:11.898353 18238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:11.899727 18223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:45:11.899891 18223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "7f0d30dbbafd495caadc58dc4979f949"
format_stamp: "Formatted at 2026-06-21 02:45:11 on dist-test-slave-hvgn"
server_key: "f96c2f7edf41caf630613d4bab9b5d29"
server_key_iv: "0709dc65f0829843ca65b277fb4a62eb"
server_key_version: "encryptionkey@0"
I20260621 02:45:11.900077 18223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:11.910825 18223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:11.914577 18223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:11.914850 18223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:11.915784 18223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:45:11.916993 18223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:45:11.917064 18223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:11.917227 18223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:45:11.917289 18223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:11.949733 18223 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:44419
I20260621 02:45:11.949776 18350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:44419 every 8 connection(s)
I20260621 02:45:11.951049 18223 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:45:11.955694 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 18223
I20260621 02:45:11.955897 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:45:11.956719 9880 external_mini_cluster.cc:1428] Setting key d3460554f56be0dc1a4b176181b17703
Jun 21 02:45:11 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009911, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:45:11.973616 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:11.953859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:38927 (local address 127.9.166.62:34841)
0621 02:45:11.954352 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:11.954359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:11.955569 (+ 1210us) server_negotiation.cc:407] Connection header received
0621 02:45:11.958120 (+ 2551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:11.958126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:11.958196 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:11.958328 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:11.960390 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:11.961130 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:11.963585 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:11.964130 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:11.964227 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:11.968246 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:11.968251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:11.968254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:11.968286 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:11.970687 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:11.971412 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:11.971420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:11.971423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:11.971553 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:11.971991 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:11.972004 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:11.972006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:11.972255 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:11.972405 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:11.973051 (+ 646us) server_negotiation.cc:299] Negotiation successful
0621 02:45:11.973264 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":230,"threads_started":1}
I20260621 02:45:11.974721 18354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:11.954534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.1:38927)
0621 02:45:11.955345 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:11.955428 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:11.957864 (+ 2436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:11.958576 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:11.958587 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:11.959353 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:11.960194 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:11.960223 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:11.961283 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:11.961291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:11.963420 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:11.963442 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:11.964380 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:11.964390 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:11.964722 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:11.965467 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:11.965502 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:11.968000 (+ 2498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:45:11.970832 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:11.970840 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:11.970856 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:11.971266 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:11.971685 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:11.971689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:11.971693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:11.971875 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:11.972380 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:45:11.972388 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:45:11.972708 (+ 320us) client_negotiation.cc:769] Sending connection context
0621 02:45:11.972960 (+ 252us) client_negotiation.cc:240] Negotiation successful
0621 02:45:11.973467 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":292,"threads_started":1}
I20260621 02:45:11.976540 18351 heartbeater.cc:347] Connected to a master server at 127.9.166.62:34841
I20260621 02:45:11.976835 18351 heartbeater.cc:464] Registering TS with master...
I20260621 02:45:11.977514 18351 heartbeater.cc:511] Master 127.9.166.62:34841 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.166.2@KRBTEST.COM; defaulting to no policy
I20260621 02:45:11.979601 18162 ts_manager.cc:194] Registered new tserver with Master: 7f0d30dbbafd495caadc58dc4979f949 (127.9.166.1:44419)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.2@KRBTEST.COM" created.
I20260621 02:45:11.981176 18162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:38927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
I20260621 02:45:12.006791 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34841
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:45:12.322388 18357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:12.322643 18357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:12.322733 18357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:12.329916 18357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:12.330013 18357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:12.330118 18357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:45:12.339586 18357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:12.340764 18357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:12.341941 18357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:12.354365 18365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:12.354379 18362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:12.354389 18363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:12.355302 18357 server_base.cc:1034] running on GCE node
I20260621 02:45:12.355960 18357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:12.356825 18357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:12.358042 18357 hybrid_clock.cc:648] HybridClock initialized: now 1782009912358001 us; error 35 us; skew 500 ppm
I20260621 02:45:12.358345 18357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:45:12 dist-test-slave-hvgn krb5kdc[18117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009912, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:45:12.361315 18357 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:45:12.362785 18357 webserver.cc:460] Webserver started at http://127.9.166.2:35493/ using document root <none> and password file <none>
I20260621 02:45:12.363503 18357 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:12.363646 18357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:12.363950 18357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:45:12.366771 18357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "01578791454349d881f4a7f9f6649c13"
format_stamp: "Formatted at 2026-06-21 02:45:12 on dist-test-slave-hvgn"
server_key: "dc3740048de0c52507f13c00231d1e56"
server_key_iv: "f1e40606678af1c3e444fc86a3025c2a"
server_key_version: "encryptionkey@0"
I20260621 02:45:12.367583 18357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "01578791454349d881f4a7f9f6649c13"
format_stamp: "Formatted at 2026-06-21 02:45:12 on dist-test-slave-hvgn"
server_key: "dc3740048de0c52507f13c00231d1e56"
server_key_iv: "f1e40606678af1c3e444fc86a3025c2a"
server_key_version: "encryptionkey@0"
I20260621 02:45:12.372396 18357 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260621 02:45:12.375761 18372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:12.377044 18357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260621 02:45:12.377221 18357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "01578791454349d881f4a7f9f6649c13"
format_stamp: "Formatted at 2026-06-21 02:45:12 on dist-test-slave-hvgn"
server_key: "dc3740048de0c52507f13c00231d1e56"
server_key_iv: "f1e40606678af1c3e444fc86a3025c2a"
server_key_version: "encryptionkey@0"
I20260621 02:45:12.377429 18357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:12.392859 18357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:12.396775 18357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:12.397068 18357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:12.398015 18357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:45:12.399372 18357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:45:12.399523 18357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260621 02:45:12.399662 18357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:45:12.399734 18357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:12.431898 18357 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:32919
I20260621 02:45:12.431983 18484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:32919 every 8 connection(s)
I20260621 02:45:12.433149 18357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:45:12.436547 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 18357
I20260621 02:45:12.436729 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:45:12.437359 9880 external_mini_cluster.cc:1428] Setting key f61d6a2ea7caef0f2ddb162a0937347c
Jun 21 02:45:12 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009912, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:45:12.453635 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:12.436085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:36417 (local address 127.9.166.62:34841)
0621 02:45:12.436271 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:12.436275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:12.437669 (+ 1394us) server_negotiation.cc:407] Connection header received
0621 02:45:12.439791 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:12.439797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:12.439886 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:12.440000 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:12.441419 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:12.442147 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:12.443777 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:12.444179 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:12.444306 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:12.448017 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:12.448022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:12.448026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:12.448056 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:12.450580 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:12.451277 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:12.451282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:12.451286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:12.451477 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:12.451902 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:12.451907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:12.451911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:12.452180 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:12.452334 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:12.453061 (+ 727us) server_negotiation.cc:299] Negotiation successful
0621 02:45:12.453287 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 02:45:12.454599 18487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:12.436689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.2:36417)
0621 02:45:12.437502 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:12.437551 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:12.439542 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:12.440225 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:12.440233 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:12.440727 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:12.441261 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:12.441277 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:12.442297 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:12.442304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:12.443633 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:12.443644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:12.444386 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:12.444394 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:12.444687 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:12.445474 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:12.445506 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:12.447797 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:45:12.450716 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:12.450725 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:12.450741 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:12.451152 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:12.451581 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:12.451587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:12.451591 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:12.451785 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:12.452305 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:45:12.452313 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:45:12.452651 (+ 338us) client_negotiation.cc:769] Sending connection context
0621 02:45:12.452958 (+ 307us) client_negotiation.cc:240] Negotiation successful
0621 02:45:12.453477 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":315,"threads_started":1}
I20260621 02:45:12.456092 18485 heartbeater.cc:347] Connected to a master server at 127.9.166.62:34841
I20260621 02:45:12.456405 18485 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.166.3@KRBTEST.COM; defaulting to no policy
I20260621 02:45:12.457147 18485 heartbeater.cc:511] Master 127.9.166.62:34841 requested a full tablet report, sending...
I20260621 02:45:12.458660 18162 ts_manager.cc:194] Registered new tserver with Master: 01578791454349d881f4a7f9f6649c13 (127.9.166.2:32919)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.3@KRBTEST.COM" created.
I20260621 02:45:12.459728 18162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:36417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
I20260621 02:45:12.482333 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34841
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:45:12.801285 18490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:12.801563 18490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:12.801718 18490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:12.808905 18490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:12.809008 18490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:12.809239 18490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:45:12.818534 18490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:34841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:12.819805 18490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:12.821055 18490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:12.833542 18496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:12.833581 18495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:12.833506 18498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:12.834275 18490 server_base.cc:1034] running on GCE node
I20260621 02:45:12.834884 18490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:12.835937 18490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:12.837172 18490 hybrid_clock.cc:648] HybridClock initialized: now 1782009912837121 us; error 46 us; skew 500 ppm
I20260621 02:45:12.837488 18490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:45:12 dist-test-slave-hvgn krb5kdc[18117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009912, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:45:12.840670 18490 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:45:12.842265 18490 webserver.cc:460] Webserver started at http://127.9.166.3:38507/ using document root <none> and password file <none>
I20260621 02:45:12.842933 18490 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:12.843061 18490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:12.843425 18490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:45:12.846230 18490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "e43c489fd82b4c7b8e57a3ced7006e86"
format_stamp: "Formatted at 2026-06-21 02:45:12 on dist-test-slave-hvgn"
server_key: "5715787e3d8b3d0ae0366c5b265b54b9"
server_key_iv: "8c96846d1db4af4949e22d05a1cb7162"
server_key_version: "encryptionkey@0"
I20260621 02:45:12.847010 18490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "e43c489fd82b4c7b8e57a3ced7006e86"
format_stamp: "Formatted at 2026-06-21 02:45:12 on dist-test-slave-hvgn"
server_key: "5715787e3d8b3d0ae0366c5b265b54b9"
server_key_iv: "8c96846d1db4af4949e22d05a1cb7162"
server_key_version: "encryptionkey@0"
I20260621 02:45:12.852016 18490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260621 02:45:12.855360 18505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:12.856748 18490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:45:12.856918 18490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "e43c489fd82b4c7b8e57a3ced7006e86"
format_stamp: "Formatted at 2026-06-21 02:45:12 on dist-test-slave-hvgn"
server_key: "5715787e3d8b3d0ae0366c5b265b54b9"
server_key_iv: "8c96846d1db4af4949e22d05a1cb7162"
server_key_version: "encryptionkey@0"
I20260621 02:45:12.857117 18490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:12.892031 18490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:12.896332 18490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:12.896617 18490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:12.897521 18490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:45:12.898772 18490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:45:12.898871 18490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:12.899024 18490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:45:12.899102 18490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:12.931908 18490 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:43313
I20260621 02:45:12.931951 18617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:43313 every 8 connection(s)
I20260621 02:45:12.933188 18490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:45:12.940966 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 18490
I20260621 02:45:12.941154 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:45:12.941599 9880 external_mini_cluster.cc:1428] Setting key 7d3f525417a11720ca1c46710c717e93
Jun 21 02:45:12 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009912, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:45:12.953823 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:12.935836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:50875 (local address 127.9.166.62:34841)
0621 02:45:12.936017 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:12.936022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:12.937358 (+ 1336us) server_negotiation.cc:407] Connection header received
0621 02:45:12.939222 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:12.939228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:12.939311 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:12.939454 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:12.941014 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:12.941877 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:12.943951 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:12.944513 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:12.944648 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:12.947845 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:12.947852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:12.947857 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:12.947897 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:12.950847 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:12.951595 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:12.951600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:12.951603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:12.951754 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:12.952153 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:12.952157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:12.952160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:12.952516 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:12.952669 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:12.953276 (+ 607us) server_negotiation.cc:299] Negotiation successful
0621 02:45:12.953481 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:45:12.954622 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:12.936420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.3:50875)
0621 02:45:12.937196 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:12.937250 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:12.938954 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:12.939680 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:12.939688 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:12.940175 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:12.940827 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:12.940851 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:12.942027 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:12.942033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:12.943812 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:12.943823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:12.944727 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:12.944736 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:12.944985 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:12.945557 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:12.945581 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:12.947654 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:45:12.950999 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:12.951005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:12.951015 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:12.951485 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:12.951862 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:12.951865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:12.951867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:12.952036 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:12.952654 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:45:12.952659 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:45:12.952917 (+ 258us) client_negotiation.cc:769] Sending connection context
0621 02:45:12.953130 (+ 213us) client_negotiation.cc:240] Negotiation successful
0621 02:45:12.953523 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":328,"threads_started":1}
I20260621 02:45:12.956086 18618 heartbeater.cc:347] Connected to a master server at 127.9.166.62:34841
I20260621 02:45:12.956386 18618 heartbeater.cc:464] Registering TS with master...
I20260621 02:45:12.957050 18618 heartbeater.cc:511] Master 127.9.166.62:34841 requested a full tablet report, sending...
I20260621 02:45:12.958374 18161 ts_manager.cc:194] Registered new tserver with Master: e43c489fd82b4c7b8e57a3ced7006e86 (127.9.166.3:43313)
I20260621 02:45:12.959199 18161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:50875
I20260621 02:45:12.969058 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:45:12.984688 18351 heartbeater.cc:503] Master 127.9.166.62:34841 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.1.keytab.
Entry for principal oryx/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.1.keytab.
WARNING: no policy specified for oryx/127.9.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.2.keytab.
Entry for principal oryx/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.2.keytab.
WARNING: no policy specified for oryx/127.9.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.3.keytab.
Entry for principal oryx/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.3.keytab.
I20260621 02:45:13.089788 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 18223
Jun 21 02:45:13 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:45:13.106356 18628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:13.090965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36830 (local address 127.9.166.1:44419)
0621 02:45:13.091739 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:13.091746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:13.091772 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:45:13.091833 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:13.091839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:13.092054 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:13.092185 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:13.093423 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:13.094425 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:13.095758 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:13.096309 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:13.096404 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:13.100679 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:13.100691 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:13.100707 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:13.100749 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:13.103533 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:13.104241 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:13.104256 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:13.104259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:13.104391 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:13.104769 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:13.104777 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:13.104779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:13.105123 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:13.105424 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:13.105741 (+ 317us) server_negotiation.cc:299] Negotiation successful
0621 02:45:13.106011 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":431,"threads_started":1}
I20260621 02:45:13.107784 18346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:13.189785 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 18223
I20260621 02:45:13.205875 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 18357
Jun 21 02:45:13 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.2@KRBTEST.COM
I20260621 02:45:13.221241 18630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:13.206770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39194 (local address 127.9.166.2:32919)
0621 02:45:13.207271 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:13.207280 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:13.207304 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:45:13.207361 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:13.207367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:13.207632 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:13.207809 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:13.209037 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:13.210017 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:13.211275 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:13.211958 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:13.212097 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:13.216050 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:13.216060 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:13.216073 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:13.216104 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:13.218636 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:13.219263 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:13.219271 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:13.219274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:13.219478 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:13.219896 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:13.219900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:13.219902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:13.220205 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:13.220423 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:13.220662 (+ 239us) server_negotiation.cc:299] Negotiation successful
0621 02:45:13.220866 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":272,"threads_started":1}
I20260621 02:45:13.222455 18480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:13.308254 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 18357
I20260621 02:45:13.323987 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 18490
Jun 21 02:45:13 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.3@KRBTEST.COM
I20260621 02:45:13.340288 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:13.324939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.9.166.3:43313)
0621 02:45:13.325568 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:13.325574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:13.325602 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:45:13.325659 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:13.325665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:13.325875 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:13.326053 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:13.327337 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:13.328458 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:13.329757 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:13.330302 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:13.330436 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:13.334677 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:13.334687 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:13.334700 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:13.334736 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:13.337735 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:13.338315 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:13.338321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:13.338324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:13.338451 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:13.338745 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:13.338748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:13.338749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:13.339068 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:13.339292 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:13.339714 (+ 422us) server_negotiation.cc:299] Negotiation successful
0621 02:45:13.339945 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":326,"threads_started":1}
I20260621 02:45:13.341568 18613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:13.423449 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 18490
I20260621 02:45:13.439605 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 18130
I20260621 02:45:13.440719 18192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:13.543923 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 18130
I20260621 02:45:13.564417 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:34841
--webserver_interface=127.9.166.62
--webserver_port=37705
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:34841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:45:13.882660 18635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:13.882929 18635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:13.883113 18635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:13.890322 18635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:45:13.890421 18635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:13.890530 18635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:13.890630 18635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:45:13.890695 18635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:45:13.900362 18635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:34841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:34841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=37705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:13.901683 18635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:13.903326 18635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:13.914074 18640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:13.915271 18641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:13.915524 18643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:13.916455 18635 server_base.cc:1034] running on GCE node
I20260621 02:45:13.917162 18635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:13.918514 18635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:13.919734 18635 hybrid_clock.cc:648] HybridClock initialized: now 1782009913919699 us; error 66 us; skew 500 ppm
I20260621 02:45:13.920055 18635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:45:13 dist-test-slave-hvgn krb5kdc[18117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009913, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:45:13.923014 18635 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:45:13.924392 18635 webserver.cc:460] Webserver started at http://127.9.166.62:37705/ using document root <none> and password file <none>
I20260621 02:45:13.925019 18635 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:13.925122 18635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:13.930179 18635 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260621 02:45:13.932902 18650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:13.934264 18635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:45:13.934439 18635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "a31555c7897d46ae92da555498d0a070"
format_stamp: "Formatted at 2026-06-21 02:45:11 on dist-test-slave-hvgn"
server_key: "d49ca6cd7c91e22bb304e471304f0b2c"
server_key_iv: "0d284dbba153a9de26300e046fce71b2"
server_key_version: "encryptionkey@0"
I20260621 02:45:13.935351 18635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:13.948392 18635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:13.952332 18635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:13.952596 18635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:13.971136 18635 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:34841
I20260621 02:45:13.971159 18701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:34841 every 8 connection(s)
I20260621 02:45:13.972608 18635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:45:13.974395 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 18635
I20260621 02:45:13.975175 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:44419
--local_ip_for_outbound_sockets=127.9.166.1
--tserver_master_addrs=127.9.166.62:34841
--webserver_port=46033
--webserver_interface=127.9.166.1
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.166.1
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:45:13.979859 18702 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:45:13.983443 18702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070: Bootstrap starting.
I20260621 02:45:14.002573 18702 log.cc:824] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070: Log is configured to *not* fsync() on all Append() calls
I20260621 02:45:14.011261 18702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:45:14.011813 18702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070: Bootstrap complete.
I20260621 02:45:14.020711 18702 raft_consensus.cc:348] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31555c7897d46ae92da555498d0a070" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34841 } }
I20260621 02:45:14.021549 18702 raft_consensus.cc:729] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a31555c7897d46ae92da555498d0a070, State: Initialized, Role: FOLLOWER
I20260621 02:45:14.022238 18702 consensus_queue.cc:260] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31555c7897d46ae92da555498d0a070" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34841 } }
I20260621 02:45:14.022485 18702 raft_consensus.cc:388] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:45:14.022604 18702 raft_consensus.cc:482] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:45:14.022790 18702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [term 1 FOLLOWER]: Advancing to term 2
I20260621 02:45:14.026429 18702 raft_consensus.cc:504] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31555c7897d46ae92da555498d0a070" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34841 } }
I20260621 02:45:14.026894 18702 leader_election.cc:302] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a31555c7897d46ae92da555498d0a070; no voters:
I20260621 02:45:14.027544 18702 leader_election.cc:288] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 02:45:14.027684 18706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 02:45:14.028766 18706 raft_consensus.cc:686] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [term 2 LEADER]: Becoming Leader. State: Replica: a31555c7897d46ae92da555498d0a070, State: Running, Role: LEADER
I20260621 02:45:14.029379 18706 consensus_queue.cc:237] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31555c7897d46ae92da555498d0a070" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34841 } }
I20260621 02:45:14.029745 18702 sys_catalog.cc:564] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:45:14.031951 18707 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a31555c7897d46ae92da555498d0a070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31555c7897d46ae92da555498d0a070" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34841 } } }
I20260621 02:45:14.032292 18707 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [sys.catalog]: This master's current role is: LEADER
I20260621 02:45:14.032755 18708 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a31555c7897d46ae92da555498d0a070. Latest consensus state: current_term: 2 leader_uuid: "a31555c7897d46ae92da555498d0a070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31555c7897d46ae92da555498d0a070" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34841 } } }
I20260621 02:45:14.033421 18708 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070 [sys.catalog]: This master's current role is: LEADER
I20260621 02:45:14.034139 18712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:45:14.041786 18712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:45:14.043540 18712 catalog_manager.cc:1259] Loaded cluster ID: 8dc39c6891f9487f8785298ad98412ae
I20260621 02:45:14.043696 18712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:45:14.048682 18712 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:45:14.050485 18712 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a31555c7897d46ae92da555498d0a070: Loaded TSK: 0
I20260621 02:45:14.052389 18712 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 02:45:14.333367 18704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:14.333668 18704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:14.333868 18704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:14.341640 18704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:14.341790 18704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:14.341971 18704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:45:14.351838 18704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.1.keytab
--principal=oryx/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:44419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=46033
--tserver_master_addrs=127.9.166.62:34841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:14.353190 18704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:14.354471 18704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:14.369946 18729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:14.370347 18731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:14.370308 18704 server_base.cc:1034] running on GCE node
W20260621 02:45:14.370012 18728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:14.371161 18704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:14.372579 18704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:14.373811 18704 hybrid_clock.cc:648] HybridClock initialized: now 1782009914373746 us; error 55 us; skew 500 ppm
I20260621 02:45:14.374080 18704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:45:14 dist-test-slave-hvgn krb5kdc[18117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009914, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:45:14.377296 18704 init.cc:377] Logged in from keytab as oryx/127.9.166.1@KRBTEST.COM (short username oryx)
I20260621 02:45:14.378831 18704 webserver.cc:460] Webserver started at http://127.9.166.1:46033/ using document root <none> and password file <none>
I20260621 02:45:14.379638 18704 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:14.379781 18704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:14.385032 18704 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 02:45:14.387759 18738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:14.389109 18704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:45:14.389285 18704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "7f0d30dbbafd495caadc58dc4979f949"
format_stamp: "Formatted at 2026-06-21 02:45:11 on dist-test-slave-hvgn"
server_key: "f96c2f7edf41caf630613d4bab9b5d29"
server_key_iv: "0709dc65f0829843ca65b277fb4a62eb"
server_key_version: "encryptionkey@0"
I20260621 02:45:14.390195 18704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:14.402426 18704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:14.406795 18704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:14.407070 18704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:14.408156 18704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:45:14.409580 18704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:45:14.409699 18704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:14.409823 18704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:45:14.409919 18704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:14.444308 18704 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:44419
I20260621 02:45:14.444429 18850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:44419 every 8 connection(s)
I20260621 02:45:14.445734 18704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:45:14.446964 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 18704
I20260621 02:45:14.447748 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:32919
--local_ip_for_outbound_sockets=127.9.166.2
--tserver_master_addrs=127.9.166.62:34841
--webserver_port=35493
--webserver_interface=127.9.166.2
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.166.2
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:45:14 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:14 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:14.467136 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:14.449171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.1:45499)
0621 02:45:14.451943 (+ 2772us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:14.451984 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:14.453728 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:14.455163 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:14.455171 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:14.455757 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:14.456419 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:14.456437 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.458855 (+ 2418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.458859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:14.460140 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:14.460152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.461661 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.461666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:14.461846 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:14.462522 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:14.462543 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:14.466246 (+ 3703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2535,"thread_start_us":2356,"threads_started":1}
W20260621 02:45:14.467856 18851 heartbeater.cc:644] Failed to heartbeat to 127.9.166.62:34841 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.166.62:34841: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
W20260621 02:45:14.468044 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:14.448892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:45499 (local address 127.9.166.62:34841)
0621 02:45:14.449590 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:14.449601 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:14.452124 (+ 2523us) server_negotiation.cc:407] Connection header received
0621 02:45:14.454271 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:14.454284 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:14.454609 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:14.455053 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:14.456617 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.458693 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.460281 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.461380 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.461565 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:14.466750 (+ 5185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.1:45499: BlockingRecv error: recv got EOF from 127.9.166.1:45499 (error 108)
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":281,"threads_started":1}
Jun 21 02:45:14 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:14 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:14.476711 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:14.468526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.1:47135)
0621 02:45:14.468763 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:14.468783 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:14.468877 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:14.469395 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:14.469398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:14.469687 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:14.469817 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:14.469824 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.471350 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.471354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:14.472263 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:14.472273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.473160 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.473164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:14.473278 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:14.473850 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:14.473866 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:14.476533 (+ 2667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129}
W20260621 02:45:14.476974 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:14.468862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:47135 (local address 127.9.166.62:34841)
0621 02:45:14.469061 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:14.469069 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:14.469090 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:45:14.469149 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:14.469171 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:14.469255 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:14.469372 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:14.469954 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.471203 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.472404 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.473028 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.473141 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:14.476765 (+ 3624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.1:47135: BlockingRecv error: recv got EOF from 127.9.166.1:47135 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 21 02:45:14 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:14 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:14.485050 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:14.477506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.1:36573)
0621 02:45:14.477650 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:14.477679 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:14.477758 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:14.478045 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:14.478048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:14.478288 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:14.478413 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:14.478421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.479821 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.479825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:14.480710 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:14.480720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.481599 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.481602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:14.481738 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:14.482262 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:14.482277 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:14.484819 (+ 2542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260621 02:45:14.485240 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:14.477565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:36573 (local address 127.9.166.62:34841)
0621 02:45:14.477714 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:14.477720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:14.477737 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:45:14.477848 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:14.477853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:14.477928 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:14.478022 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:14.478548 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.479668 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.480839 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.481472 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.481588 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:14.485008 (+ 3420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.1:36573: BlockingRecv error: recv got EOF from 127.9.166.1:36573 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260621 02:45:14.485397 18851 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 02:45:14.774706 18855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:14.774961 18855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:14.775125 18855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:14.782464 18855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:14.782563 18855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:14.782759 18855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:45:14.792459 18855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.2.keytab
--principal=oryx/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:32919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=35493
--tserver_master_addrs=127.9.166.62:34841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:14.793875 18855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:14.795369 18855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:14.808197 18860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:14.808218 18861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:14.808878 18863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:14.809387 18855 server_base.cc:1034] running on GCE node
I20260621 02:45:14.809962 18855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:14.810858 18855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:14.812079 18855 hybrid_clock.cc:648] HybridClock initialized: now 1782009914812031 us; error 60 us; skew 500 ppm
I20260621 02:45:14.812414 18855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:45:14 dist-test-slave-hvgn krb5kdc[18117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009914, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:45:14.815361 18855 init.cc:377] Logged in from keytab as oryx/127.9.166.2@KRBTEST.COM (short username oryx)
I20260621 02:45:14.816731 18855 webserver.cc:460] Webserver started at http://127.9.166.2:35493/ using document root <none> and password file <none>
I20260621 02:45:14.817414 18855 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:14.817565 18855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:14.822928 18855 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:45:14.825606 18870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:14.826800 18855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260621 02:45:14.826974 18855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "01578791454349d881f4a7f9f6649c13"
format_stamp: "Formatted at 2026-06-21 02:45:12 on dist-test-slave-hvgn"
server_key: "dc3740048de0c52507f13c00231d1e56"
server_key_iv: "f1e40606678af1c3e444fc86a3025c2a"
server_key_version: "encryptionkey@0"
I20260621 02:45:14.827998 18855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:14.848181 18855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:14.852452 18855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:14.852730 18855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:14.853693 18855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:45:14.855046 18855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:45:14.855180 18855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:14.855319 18855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:45:14.855434 18855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:14.889216 18855 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:32919
I20260621 02:45:14.889304 18982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:32919 every 8 connection(s)
I20260621 02:45:14.890664 18855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:45:14.891682 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 18855
I20260621 02:45:14.892293 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:43313
--local_ip_for_outbound_sockets=127.9.166.3
--tserver_master_addrs=127.9.166.62:34841
--webserver_port=38507
--webserver_interface=127.9.166.3
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.166.3
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:45:14 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:14 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 02:45:14.908661 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:14.893301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:56051 (local address 127.9.166.62:34841)
0621 02:45:14.893735 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:14.893741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:14.894999 (+ 1258us) server_negotiation.cc:407] Connection header received
0621 02:45:14.896916 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:14.896923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:14.897017 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:14.897137 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:14.898591 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.899792 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.902300 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.902703 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.902791 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:14.908418 (+ 5627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.2:56051: BlockingRecv error: recv got EOF from 127.9.166.2:56051 (error 108)
Metrics: {"server-negotiator.queue_time_us":290}
I20260621 02:45:14.909387 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:14.893941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.2:56051)
0621 02:45:14.894822 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:14.894869 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:14.896634 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:14.897335 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:14.897343 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:14.897865 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:14.898404 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:14.898421 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.899957 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.899965 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:14.902162 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:14.902179 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.902928 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.902937 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:14.903264 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:14.903993 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:14.904030 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:14.908230 (+ 4200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":379,"threads_started":1}
W20260621 02:45:14.910136 18983 heartbeater.cc:644] Failed to heartbeat to 127.9.166.62:34841 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.166.62:34841: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Jun 21 02:45:14 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:14 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:14.918184 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:14.910766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.2:42305)
0621 02:45:14.911012 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:14.911030 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:14.911126 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:14.911505 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:14.911509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:14.911812 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:14.911938 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:14.911945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.912893 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.912900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:14.913805 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:14.913815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.914497 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.914502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:14.914662 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:14.915254 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:14.915271 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:14.917913 (+ 2642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132}
W20260621 02:45:14.918347 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:14.910981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:42305 (local address 127.9.166.62:34841)
0621 02:45:14.911187 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:14.911192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:14.911211 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:45:14.911269 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:14.911272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:14.911334 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:14.911467 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:14.912071 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.912759 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.913943 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.914340 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.914475 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:14.918120 (+ 3645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.2:42305: BlockingRecv error: recv got EOF from 127.9.166.2:42305 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Jun 21 02:45:14 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:14 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:14.926378 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:14.918881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.2:46393)
0621 02:45:14.919043 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:14.919061 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:14.919158 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:14.919517 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:14.919520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:14.919757 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:14.919886 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:14.919892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.921188 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.921191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:14.922078 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:14.922087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.922972 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.922987 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:14.923122 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:14.923682 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:14.923698 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:14.926192 (+ 2494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260621 02:45:14.926633 18983 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 02:45:14.926869 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:14.919048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:46393 (local address 127.9.166.62:34841)
0621 02:45:14.919233 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:14.919239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:14.919257 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:45:14.919302 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:14.919307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:14.919370 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:14.919484 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:14.920005 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.921068 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.922205 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:14.922846 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:14.922960 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:14.926406 (+ 3446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.2:46393: BlockingRecv error: recv got EOF from 127.9.166.2:46393 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260621 02:45:15.214565 18986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:15.214826 18986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:15.214919 18986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:15.222069 18986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:15.222178 18986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:15.222293 18986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:45:15.231819 18986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/krb5kdc/oryx_127.9.166.3.keytab
--principal=oryx/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:43313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=38507
--tserver_master_addrs=127.9.166.62:34841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:15.232976 18986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:15.234176 18986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:15.246698 18991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:15.246629 18994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:15.246758 18992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:15.247579 18986 server_base.cc:1034] running on GCE node
I20260621 02:45:15.248076 18986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:15.248889 18986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:15.250113 18986 hybrid_clock.cc:648] HybridClock initialized: now 1782009915250039 us; error 59 us; skew 500 ppm
I20260621 02:45:15.250411 18986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:45:15 dist-test-slave-hvgn krb5kdc[18117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009915, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:45:15.253409 18986 init.cc:377] Logged in from keytab as oryx/127.9.166.3@KRBTEST.COM (short username oryx)
I20260621 02:45:15.254873 18986 webserver.cc:460] Webserver started at http://127.9.166.3:38507/ using document root <none> and password file <none>
I20260621 02:45:15.255643 18986 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:15.255802 18986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:15.260792 18986 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260621 02:45:15.263203 19001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:15.264528 18986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:45:15.264698 18986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "e43c489fd82b4c7b8e57a3ced7006e86"
format_stamp: "Formatted at 2026-06-21 02:45:12 on dist-test-slave-hvgn"
server_key: "5715787e3d8b3d0ae0366c5b265b54b9"
server_key_iv: "8c96846d1db4af4949e22d05a1cb7162"
server_key_version: "encryptionkey@0"
I20260621 02:45:15.265645 18986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:15.289268 18986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:15.293013 18986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:15.293279 18986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:15.294229 18986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:45:15.295624 18986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:45:15.295707 18986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:15.295786 18986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:45:15.295840 18986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:15.326972 18986 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:43313
I20260621 02:45:15.327066 19113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:43313 every 8 connection(s)
I20260621 02:45:15.328409 18986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:45:15.335317 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 18986
Jun 21 02:45:15 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:15 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
W20260621 02:45:15.345705 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:15.330954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:57851 (local address 127.9.166.62:34841)
0621 02:45:15.331142 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:15.331146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:15.332494 (+ 1348us) server_negotiation.cc:407] Connection header received
0621 02:45:15.334341 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:15.334347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:15.334434 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:15.334558 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:15.336310 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.337375 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.338991 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.339620 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.339720 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:15.345403 (+ 5683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.3:57851: BlockingRecv error: recv got EOF from 127.9.166.3:57851 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:45:15.346170 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:15.331553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.3:57851)
0621 02:45:15.332294 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:15.332339 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:15.334062 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:15.334794 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:15.334802 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:15.335397 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:15.336149 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:15.336169 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.337547 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.337553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:15.338845 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:15.338861 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.339835 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.339843 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:15.340099 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:15.340822 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:15.340847 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:15.345167 (+ 4320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":286,"threads_started":1}
W20260621 02:45:15.346954 19114 heartbeater.cc:644] Failed to heartbeat to 127.9.166.62:34841 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.166.62:34841: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
I20260621 02:45:15.351794 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:15.336403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42486 (local address 127.9.166.62:34841)
0621 02:45:15.337153 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:15.337157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:15.337172 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:45:15.337209 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:15.337212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:15.337266 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:15.337354 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:15.338902 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.340064 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.341493 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.342090 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.342183 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:15.345351 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:15.345362 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:15.345383 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:15.345434 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:15.348712 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:15.349336 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:15.349351 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:15.349358 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:15.349572 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:15.349930 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:15.349937 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:15.349940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:15.350326 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:15.350628 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:15.351034 (+ 406us) server_negotiation.cc:299] Negotiation successful
0621 02:45:15.351535 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":549,"threads_started":1}
Jun 21 02:45:15 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:15 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:15.354799 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:15.347544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.3:60463)
0621 02:45:15.347749 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:15.347766 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:15.347857 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:15.348142 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:15.348147 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:15.348461 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:15.348579 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:15.348586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.349580 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.349589 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:15.350436 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:15.350446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.351163 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.351169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:15.351338 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:15.351899 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:15.351927 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:15.354614 (+ 2687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260621 02:45:15.355160 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:15.347639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:60463 (local address 127.9.166.62:34841)
0621 02:45:15.347813 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:15.347819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:15.347835 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:45:15.347959 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:15.347964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:15.348023 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:15.348124 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:15.348714 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.349431 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.350574 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.351025 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.351159 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:15.354827 (+ 3668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.3:60463: BlockingRecv error: recv got EOF from 127.9.166.3:60463 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 21 02:45:15 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:15 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:15.361971 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:15.355514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.3:45037)
0621 02:45:15.355658 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:15.355679 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:15.355767 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:15.355979 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:15.355982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:15.356226 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:15.356352 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:15.356358 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.357274 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.357277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:15.358123 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:15.358133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.358754 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.358758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:15.358888 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:15.359315 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:15.359329 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:15.361767 (+ 2438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260621 02:45:15.362110 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:15.355590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:45037 (local address 127.9.166.62:34841)
0621 02:45:15.355798 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:15.355802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:15.355815 (+ 13us) server_negotiation.cc:407] Connection header received
0621 02:45:15.355849 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:15.355852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:15.355897 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:15.355962 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:15.356458 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.357169 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.358252 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.358623 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.358733 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:15.361923 (+ 3190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.3:45037: BlockingRecv error: recv got EOF from 127.9.166.3:45037 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
W20260621 02:45:15.362272 19114 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 21 02:45:15 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:15 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:15.494447 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:15.485963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.1:45155)
0621 02:45:15.486457 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:15.486476 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:15.486631 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:15.486994 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:15.486999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:15.487503 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:15.487684 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:15.487693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.488710 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.488718 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:15.489678 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:15.489689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.490433 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.490438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:15.490597 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:15.491191 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:15.491211 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:15.494119 (+ 2908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":267,"threads_started":1}
W20260621 02:45:15.494541 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:15.486026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:45155 (local address 127.9.166.62:34841)
0621 02:45:15.486270 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:15.486278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:15.486563 (+ 285us) server_negotiation.cc:407] Connection header received
0621 02:45:15.486786 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:15.486791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:15.486859 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:15.486960 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:15.487846 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.488564 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.489827 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.490257 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.490381 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:15.494325 (+ 3944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.1:45155: BlockingRecv error: recv got EOF from 127.9.166.1:45155 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 21 02:45:15 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:15 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:15.935735 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:15.927204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.2:50003)
0621 02:45:15.927716 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:15.927731 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:15.927839 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:15.928194 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:15.928197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:15.928636 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:15.928821 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:15.928829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.929944 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.929947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:15.930979 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:15.930991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.931757 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.931764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:15.931917 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:15.932492 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:15.932510 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:15.935385 (+ 2875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":281,"threads_started":1}
W20260621 02:45:15.935918 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:15.927244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:50003 (local address 127.9.166.62:34841)
0621 02:45:15.927496 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:15.927501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:15.927800 (+ 299us) server_negotiation.cc:407] Connection header received
0621 02:45:15.927959 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:15.927963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:15.928047 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:15.928145 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:15.928963 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.929789 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.931114 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:15.931586 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:15.931678 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:15.935638 (+ 3960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.2:50003: BlockingRecv error: recv got EOF from 127.9.166.2:50003 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 21 02:45:16 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:16 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:16.371239 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:16.362841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.3:32841)
0621 02:45:16.363326 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:16.363348 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:16.363562 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:16.363907 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:16.363912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:16.364395 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:16.364586 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:16.364594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:16.365605 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:16.365611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:16.366569 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:16.366580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:16.367302 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:16.367307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:16.367518 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:16.368134 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:16.368152 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:16.370908 (+ 2756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":255,"threads_started":1}
W20260621 02:45:16.371349 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:16.362910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:32841 (local address 127.9.166.62:34841)
0621 02:45:16.363145 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:16.363153 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:16.363466 (+ 313us) server_negotiation.cc:407] Connection header received
0621 02:45:16.363689 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:16.363700 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:16.363761 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:16.363864 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:16.364729 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:16.365467 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:16.366708 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:16.367108 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:16.367257 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:16.371114 (+ 3857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.3:32841: BlockingRecv error: recv got EOF from 127.9.166.3:32841 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 21 02:45:16 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:16 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:16.503602 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:16.495320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.1:37971)
0621 02:45:16.495831 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:16.495850 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:16.496005 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:16.496363 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:16.496368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:16.496783 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:16.496917 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:16.496925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:16.497969 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:16.497976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:16.498990 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:16.499002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:16.499734 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:16.499739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:16.499904 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:16.500432 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:16.500448 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:16.503282 (+ 2834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":279,"threads_started":1}
W20260621 02:45:16.503690 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:16.495386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:37971 (local address 127.9.166.62:34841)
0621 02:45:16.495628 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:16.495636 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:16.495937 (+ 301us) server_negotiation.cc:407] Connection header received
0621 02:45:16.496144 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:16.496148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:16.496206 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:16.496307 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:16.497061 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:16.497834 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:16.499128 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:16.499575 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:16.499720 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:16.503492 (+ 3772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.1:37971: BlockingRecv error: recv got EOF from 127.9.166.1:37971 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 21 02:45:16 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:16 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:16.949800 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:16.936735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.2:46353)
0621 02:45:16.937298 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:16.937312 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:16.937469 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:16.937835 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:16.937839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:16.938267 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:16.938407 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:16.938415 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:16.939793 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:16.939800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:16.940819 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:16.940830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:16.941526 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:16.941531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:16.941698 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:16.946735 (+ 5037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:16.946759 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:16.949556 (+ 2797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":321,"threads_started":1}
W20260621 02:45:16.950088 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:16.936802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:46353 (local address 127.9.166.62:34841)
0621 02:45:16.937021 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:16.937030 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:16.937397 (+ 367us) server_negotiation.cc:407] Connection header received
0621 02:45:16.937602 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:16.937605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:16.937665 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:16.937793 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:16.938548 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:16.939666 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:16.940960 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:16.941378 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:16.941511 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:16.949770 (+ 8259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.2:46353: BlockingRecv error: recv got EOF from 127.9.166.2:46353 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 21 02:45:17 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:17 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:17.380195 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:17.372123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.3:38455)
0621 02:45:17.372634 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:17.372651 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:17.372759 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:17.373046 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:17.373049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:17.373443 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:17.373581 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:17.373589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:17.374603 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:17.374606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:17.375556 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:17.375568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:17.376266 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:17.376271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:17.376446 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:17.377083 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:17.377101 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:17.379957 (+ 2856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":299,"threads_started":1}
W20260621 02:45:17.380364 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:17.372287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:38455 (local address 127.9.166.62:34841)
0621 02:45:17.372477 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:17.372481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:17.372725 (+ 244us) server_negotiation.cc:407] Connection header received
0621 02:45:17.372855 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:17.372859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:17.372924 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:17.373002 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:17.373726 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:17.374472 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:17.375679 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:17.376105 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:17.376238 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:17.380122 (+ 3884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.3:38455: BlockingRecv error: recv got EOF from 127.9.166.3:38455 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 21 02:45:17 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:17 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:17.513011 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:17.504629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.1:56327)
0621 02:45:17.505089 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:17.505109 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:17.505223 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:17.505681 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:17.505685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:17.506159 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:17.506372 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:17.506386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:17.507831 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:17.507836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:17.508725 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:17.508737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:17.509374 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:17.509378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:17.509507 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:17.510068 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:17.510083 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:17.512767 (+ 2684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":257,"threads_started":1}
W20260621 02:45:17.513249 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:17.504684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:56327 (local address 127.9.166.62:34841)
0621 02:45:17.504910 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:17.504917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:17.505197 (+ 280us) server_negotiation.cc:407] Connection header received
0621 02:45:17.505331 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:17.505335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:17.505435 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:17.505517 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:17.506549 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:17.507698 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:17.508864 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:17.509236 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:17.509330 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:17.512983 (+ 3653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.1:56327: BlockingRecv error: recv got EOF from 127.9.166.1:56327 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 21 02:45:17 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:17 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:17.958949 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:17.950713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.2:60843)
0621 02:45:17.951178 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:17.951195 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:17.951307 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:17.951743 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:17.951747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:17.952102 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:17.952279 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:17.952288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:17.953311 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:17.953315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:17.954223 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:17.954234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:17.954867 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:17.954871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:17.954992 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:17.955735 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:17.955752 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:17.958591 (+ 2839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":246,"threads_started":1}
W20260621 02:45:17.959087 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:17.950810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:60843 (local address 127.9.166.62:34841)
0621 02:45:17.951067 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:17.951074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:17.951277 (+ 203us) server_negotiation.cc:407] Connection header received
0621 02:45:17.951469 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:17.951473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:17.951534 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:17.951614 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:17.952421 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:17.953189 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:17.954349 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:17.954739 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:17.954860 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:17.958793 (+ 3933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.2:60843: BlockingRecv error: recv got EOF from 127.9.166.2:60843 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Jun 21 02:45:18 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:18 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:18.389228 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:18.381197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.3:38581)
0621 02:45:18.381644 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:18.381660 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:18.381773 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:18.382083 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:18.382087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:18.382547 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:18.382774 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:18.382787 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:18.383950 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:18.383956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:18.384932 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:18.384943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:18.385594 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:18.385598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:18.385717 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:18.386269 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:18.386292 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:18.388990 (+ 2698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":245,"threads_started":1}
W20260621 02:45:18.389385 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:18.381257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:38581 (local address 127.9.166.62:34841)
0621 02:45:18.381505 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:18.381513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:18.381745 (+ 232us) server_negotiation.cc:407] Connection header received
0621 02:45:18.381883 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:18.381887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:18.381959 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:18.382036 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:18.382932 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:18.383792 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:18.385050 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:18.385463 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:18.385561 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:18.389177 (+ 3616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.3:38581: BlockingRecv error: recv got EOF from 127.9.166.3:38581 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 21 02:45:18 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:18 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:18.521865 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:18.513864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.1:56181)
0621 02:45:18.514311 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:18.514330 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:18.514441 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:18.514763 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:18.514766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:18.515127 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:18.515264 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:18.515272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:18.516233 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:18.516237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:18.517238 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:18.517249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:18.517886 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:18.517890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:18.518020 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:18.518650 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:18.518668 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:18.521506 (+ 2838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":250,"threads_started":1}
W20260621 02:45:18.521935 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:18.514006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:56181 (local address 127.9.166.62:34841)
0621 02:45:18.514256 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:18.514265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:18.514406 (+ 141us) server_negotiation.cc:407] Connection header received
0621 02:45:18.514539 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:18.514542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:18.514599 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:18.514675 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:18.515415 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:18.516111 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:18.517376 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:18.517766 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:18.517917 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:18.521698 (+ 3781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.1:56181: BlockingRecv error: recv got EOF from 127.9.166.1:56181 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Jun 21 02:45:18 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:18 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:18.968438 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:18.959969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.2:55629)
0621 02:45:18.960502 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:18.960519 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:18.960636 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:18.960931 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:18.960934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:18.961284 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:18.961450 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:18.961459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:18.962459 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:18.962463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:18.963491 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:18.963504 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:18.964214 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:18.964220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:18.964356 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:18.965017 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:18.965035 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:18.968103 (+ 3068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":296,"threads_started":1}
W20260621 02:45:18.968524 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:18.960078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:55629 (local address 127.9.166.62:34841)
0621 02:45:18.960244 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:18.960248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:18.960595 (+ 347us) server_negotiation.cc:407] Connection header received
0621 02:45:18.960735 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:18.960739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:18.960799 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:18.960887 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:18.961570 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:18.962307 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:18.963650 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:18.964067 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:18.964173 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:18.968294 (+ 4121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.2:55629: BlockingRecv error: recv got EOF from 127.9.166.2:55629 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 21 02:45:19 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:19 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:19.399065 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:19.390353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.3:41865)
0621 02:45:19.390923 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:19.390938 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:19.391063 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:19.391433 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:19.391447 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:19.391812 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:19.391955 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:19.391963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:19.392987 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:19.393011 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:19.394055 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:19.394066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:19.394724 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:19.394728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:19.394855 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:19.395572 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:19.395610 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:19.398829 (+ 3219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":328,"threads_started":1}
W20260621 02:45:19.399240 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:19.390387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:41865 (local address 127.9.166.62:34841)
0621 02:45:19.390569 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:19.390573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:19.391030 (+ 457us) server_negotiation.cc:407] Connection header received
0621 02:45:19.391188 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:19.391192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:19.391253 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:19.391338 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:19.392090 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:19.392870 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:19.394186 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:19.394604 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:19.394697 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:19.399011 (+ 4314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.3:41865: BlockingRecv error: recv got EOF from 127.9.166.3:41865 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 21 02:45:19 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:19 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:19.532477 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:19.522930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.1:50287)
0621 02:45:19.523598 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:19.523619 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:19.523786 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:19.524144 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:19.524149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:19.524600 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:19.524758 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:19.524767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:19.526025 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:19.526033 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:19.526974 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:19.526985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:19.527828 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:19.527834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:19.528008 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:19.528750 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:19.528779 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:19.532099 (+ 3320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":353,"threads_started":1}
W20260621 02:45:19.532541 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:19.523017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:50287 (local address 127.9.166.62:34841)
0621 02:45:19.523350 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:19.523357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:19.523710 (+ 353us) server_negotiation.cc:407] Connection header received
0621 02:45:19.523934 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:19.523938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:19.524000 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:19.524108 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:19.524911 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:19.525871 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:19.527149 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:19.527679 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:19.527804 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:19.532285 (+ 4481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.1:50287: BlockingRecv error: recv got EOF from 127.9.166.1:50287 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jun 21 02:45:19 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:19 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:19.979107 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:19.969437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.2:41269)
0621 02:45:19.969979 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:19.969995 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:19.970149 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:19.970510 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:19.970515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:19.971083 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:19.971326 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:19.971343 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:19.972812 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:19.972818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:19.974281 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:19.974294 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:19.974979 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:19.974985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:19.975153 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:19.975767 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:19.975789 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:19.978877 (+ 3088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":273,"threads_started":1}
W20260621 02:45:19.979411 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:19.969685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:41269 (local address 127.9.166.62:34841)
0621 02:45:19.969877 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:19.969883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:19.970097 (+ 214us) server_negotiation.cc:407] Connection header received
0621 02:45:19.970304 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:19.970308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:19.970374 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:19.970478 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:19.971493 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:19.972658 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:19.974418 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:19.974848 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:19.974937 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:19.979125 (+ 4188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.2:41269: BlockingRecv error: recv got EOF from 127.9.166.2:41269 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 21 02:45:20 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:20 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:20.407828 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:20.400001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.3:50515)
0621 02:45:20.400454 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:20.400472 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:20.400584 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:20.400859 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:20.400862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:20.401261 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:20.401419 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:20.401427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:20.402389 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:20.402392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:20.403295 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:20.403306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:20.404029 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:20.404034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:20.404160 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:20.404829 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:20.404847 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:20.407591 (+ 2744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":223,"threads_started":1}
W20260621 02:45:20.407996 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:20.400050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:50515 (local address 127.9.166.62:34841)
0621 02:45:20.400268 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:20.400275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:20.400556 (+ 281us) server_negotiation.cc:407] Connection header received
0621 02:45:20.400680 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:20.400684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:20.400744 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:20.400820 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:20.401544 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:20.402254 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:20.403466 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:20.403869 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:20.403959 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:20.407772 (+ 3813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.3:50515: BlockingRecv error: recv got EOF from 127.9.166.3:50515 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 21 02:45:20 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:20 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:20.542081 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:20.533380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.1:45855)
0621 02:45:20.533935 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:20.533968 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:20.534131 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:20.534462 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:20.534466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:20.534871 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:20.535036 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:20.535045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:20.536269 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:20.536276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:20.537256 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:20.537268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:20.537971 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:20.537977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:20.538151 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:20.538791 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:20.538808 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:20.541756 (+ 2948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":321,"threads_started":1}
W20260621 02:45:20.542191 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:20.533512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:45855 (local address 127.9.166.62:34841)
0621 02:45:20.533755 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:20.533763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:20.534056 (+ 293us) server_negotiation.cc:407] Connection header received
0621 02:45:20.534255 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:20.534262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:20.534331 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:20.534433 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:20.535178 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:20.536139 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:20.537404 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:20.537817 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:20.537940 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:20.541946 (+ 4006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.1:45855: BlockingRecv error: recv got EOF from 127.9.166.1:45855 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 21 02:45:20 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:20 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:20.988147 19135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:20.980152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.2:50025)
0621 02:45:20.980799 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:20.980815 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:20.980966 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:20.981301 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:20.981306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:20.981696 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:20.981841 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:20.981849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:20.982857 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:20.982862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:20.983798 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:20.983809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:20.984503 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:20.984510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:20.984688 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:20.985267 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:20.985284 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:20.987938 (+ 2654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":348,"threads_started":1}
W20260621 02:45:20.988427 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:20.980298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:50025 (local address 127.9.166.62:34841)
0621 02:45:20.980496 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:20.980503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:20.980897 (+ 394us) server_negotiation.cc:407] Connection header received
0621 02:45:20.981083 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:20.981088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:20.981158 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:20.981261 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:20.981991 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:20.982719 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:20.983941 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:20.984340 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:20.984464 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:20.988178 (+ 3714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.2:50025: BlockingRecv error: recv got EOF from 127.9.166.2:50025 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 21 02:45:21 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:21 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:21.418865 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:21.408875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.3:33323)
0621 02:45:21.409410 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:21.409425 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:21.409533 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:21.409928 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:21.409931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:21.410305 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:21.410508 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:21.410521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:21.412263 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:21.412267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:21.413137 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:21.413148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:21.414386 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:21.414400 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:21.414583 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:21.415318 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:21.415344 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:21.418633 (+ 3289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":304,"threads_started":1}
W20260621 02:45:21.419269 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:21.408946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:33323 (local address 127.9.166.62:34841)
0621 02:45:21.409177 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:21.409186 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:21.409504 (+ 318us) server_negotiation.cc:407] Connection header received
0621 02:45:21.409647 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:21.409655 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:21.409767 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:21.409896 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:21.410663 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:21.412113 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:21.413286 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:21.414168 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:21.414334 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:21.418955 (+ 4621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.3:33323: BlockingRecv error: recv got EOF from 127.9.166.3:33323 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 21 02:45:21 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:21 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:21.551961 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:21.543150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.1:39233)
0621 02:45:21.543780 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:21.543800 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:21.544050 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:21.544397 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:21.544402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:21.544820 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:21.545006 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:21.545015 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:21.546020 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:21.546026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:21.547140 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:21.547156 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:21.548234 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:21.548239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:21.548366 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:21.548946 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:21.548962 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:21.551729 (+ 2767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":299,"threads_started":1}
W20260621 02:45:21.552141 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:21.543219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:39233 (local address 127.9.166.62:34841)
0621 02:45:21.543492 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:21.543501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:21.543897 (+ 396us) server_negotiation.cc:407] Connection header received
0621 02:45:21.544179 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:21.544187 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:21.544250 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:21.544369 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:21.545150 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:21.545883 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:21.547324 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:21.548084 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:21.548186 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:21.551932 (+ 3746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.1:39233: BlockingRecv error: recv got EOF from 127.9.166.1:39233 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 21 02:45:21 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:21 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:22.001942 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:21.989043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.2:37821)
0621 02:45:21.989612 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:21.989628 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:21.989744 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:21.990035 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:21.990038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:21.990402 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:21.990570 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:21.990579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:21.991628 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:21.991632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:21.992555 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:21.992565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:21.993216 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:21.993220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:21.993330 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:21.998836 (+ 5506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:21.998854 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:22.001694 (+ 2840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":254,"threads_started":1}
W20260621 02:45:22.002174 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:21.989122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:37821 (local address 127.9.166.62:34841)
0621 02:45:21.989315 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:21.989319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:21.989696 (+ 377us) server_negotiation.cc:407] Connection header received
0621 02:45:21.989839 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:21.989843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:21.989913 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:21.990015 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:21.990693 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:21.991503 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:21.992674 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:21.993092 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:21.993179 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:22.001871 (+ 8692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.2:37821: BlockingRecv error: recv got EOF from 127.9.166.2:37821 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Jun 21 02:45:22 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:22 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:22.428100 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:22.419782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.3:51601)
0621 02:45:22.420429 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:22.420446 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:22.420601 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:22.420963 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:22.420968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:22.421380 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:22.421539 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:22.421548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:22.422573 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:22.422580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:22.423531 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:22.423543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:22.424277 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:22.424283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:22.424450 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:22.424999 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:22.425016 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:22.427833 (+ 2817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":321,"threads_started":1}
W20260621 02:45:22.428175 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:22.420017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:51601 (local address 127.9.166.62:34841)
0621 02:45:22.420221 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:22.420226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:22.420535 (+ 309us) server_negotiation.cc:407] Connection header received
0621 02:45:22.420738 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:22.420743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:22.420816 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:22.420917 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:22.421700 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:22.422435 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:22.423679 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:22.424126 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:22.424260 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:22.428005 (+ 3745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.3:51601: BlockingRecv error: recv got EOF from 127.9.166.3:51601 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 21 02:45:22 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:22 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:22.561043 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:22.552960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.1:42109)
0621 02:45:22.553563 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:22.553584 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:22.553708 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:22.554009 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:22.554012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:22.554408 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:22.554548 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:22.554556 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:22.555540 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:22.555544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:22.556457 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:22.556467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:22.557320 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:22.557325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:22.557448 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:22.558047 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:22.558064 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:22.560806 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":366,"threads_started":1}
W20260621 02:45:22.561275 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:22.553098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:42109 (local address 127.9.166.62:34841)
0621 02:45:22.553487 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:22.553493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:22.553689 (+ 196us) server_negotiation.cc:407] Connection header received
0621 02:45:22.553814 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:22.553818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:22.553879 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:22.553963 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:22.554671 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:22.555416 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:22.556631 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:22.557144 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:22.557252 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:22.561002 (+ 3750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.1:42109: BlockingRecv error: recv got EOF from 127.9.166.1:42109 (error 108)
Metrics: {"server-negotiator.queue_time_us":222}
Jun 21 02:45:23 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:23 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:23.011610 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:23.002831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.2:34513)
0621 02:45:23.003439 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:23.003455 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:23.003613 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:23.003952 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:23.003957 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:23.004380 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:23.004526 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:23.004534 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:23.005709 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:23.005714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:23.006661 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:23.006671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:23.007475 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:23.007481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:23.007641 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:23.008278 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:23.008296 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:23.011309 (+ 3013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":299,"threads_started":1}
W20260621 02:45:23.011823 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:23.002984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:34513 (local address 127.9.166.62:34841)
0621 02:45:23.003227 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:23.003234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:23.003546 (+ 312us) server_negotiation.cc:407] Connection header received
0621 02:45:23.003741 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:23.003746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:23.003816 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:23.003919 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:23.004671 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:23.005559 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:23.006810 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:23.007246 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:23.007395 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:23.011539 (+ 4144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.2:34513: BlockingRecv error: recv got EOF from 127.9.166.2:34513 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 21 02:45:23 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:23 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:23.437491 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:23.429035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.3:49707)
0621 02:45:23.429573 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:23.429589 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:23.429695 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:23.430051 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:23.430055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:23.430507 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:23.430653 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:23.430661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:23.431910 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:23.431916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:23.432864 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:23.432876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:23.433551 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:23.433556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:23.433667 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:23.434281 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:23.434298 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:23.437263 (+ 2965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":285,"threads_started":1}
W20260621 02:45:23.437814 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:23.429120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:49707 (local address 127.9.166.62:34841)
0621 02:45:23.429310 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:23.429315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:23.429659 (+ 344us) server_negotiation.cc:407] Connection header received
0621 02:45:23.429789 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:23.429793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:23.429853 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:23.429945 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:23.430779 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:23.431703 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:23.432996 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:23.433398 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:23.433496 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:23.437546 (+ 4050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.3:49707: BlockingRecv error: recv got EOF from 127.9.166.3:49707 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 21 02:45:23 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:23 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:23.570255 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:23.562086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.1:57933)
0621 02:45:23.562640 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:23.562662 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:23.562777 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:23.563100 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:23.563104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:23.563522 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:23.563687 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:23.563696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:23.564738 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:23.564742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:23.565685 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:23.565696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:23.566401 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:23.566407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:23.566614 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:23.567158 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:23.567174 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:23.570014 (+ 2840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":301,"threads_started":1}
W20260621 02:45:23.570482 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:23.562132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:57933 (local address 127.9.166.62:34841)
0621 02:45:23.562317 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:23.562322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:23.562737 (+ 415us) server_negotiation.cc:407] Connection header received
0621 02:45:23.562876 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:23.562879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:23.562958 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:23.563043 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:23.563852 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:23.564619 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:23.565836 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:23.566250 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:23.566405 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:23.570235 (+ 3830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.1:57933: BlockingRecv error: recv got EOF from 127.9.166.1:57933 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 21 02:45:24 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:24 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:24.021216 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:24.012666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.2:49757)
0621 02:45:24.013224 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:24.013241 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:24.013354 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:24.013669 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:24.013672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:24.014037 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:24.014210 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:24.014219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:24.015283 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:24.015288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:24.016749 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:24.016766 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:24.017425 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:24.017430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:24.017549 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:24.018181 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:24.018198 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:24.020955 (+ 2757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":312,"threads_started":1}
W20260621 02:45:24.021368 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:24.012758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:49757 (local address 127.9.166.62:34841)
0621 02:45:24.012931 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:24.012935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:24.013318 (+ 383us) server_negotiation.cc:407] Connection header received
0621 02:45:24.013451 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:24.013455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:24.013517 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:24.013602 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:24.014329 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:24.015096 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:24.016884 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:24.017296 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:24.017389 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:24.021144 (+ 3755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.2:49757: BlockingRecv error: recv got EOF from 127.9.166.2:49757 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 21 02:45:24 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:24 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:24.446995 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:24.438362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.3:34809)
0621 02:45:24.438870 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:24.438885 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:24.439033 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:24.439450 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:24.439455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:24.439941 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:24.440102 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:24.440110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:24.441194 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:24.441198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:24.442445 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:24.442456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:24.443217 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:24.443225 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:24.443421 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:24.443983 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:24.444000 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:24.446756 (+ 2756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":285,"threads_started":1}
W20260621 02:45:24.447119 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:24.438419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:34809 (local address 127.9.166.62:34841)
0621 02:45:24.438648 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:24.438656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:24.438969 (+ 313us) server_negotiation.cc:407] Connection header received
0621 02:45:24.439168 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:24.439175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:24.439255 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:24.439355 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:24.440250 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:24.441040 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:24.442563 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:24.443049 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:24.443175 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:24.446918 (+ 3743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.3:34809: BlockingRecv error: recv got EOF from 127.9.166.3:34809 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 21 02:45:24 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:24 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:24.579423 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:24.571287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.1:60707)
0621 02:45:24.571860 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:24.571892 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:24.572040 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:24.572419 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:24.572423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:24.572842 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:24.572980 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:24.572988 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:24.574019 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:24.574025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:24.574953 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:24.574963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:24.575762 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:24.575767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:24.575928 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:24.576499 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:24.576516 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:24.579142 (+ 2626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":314,"threads_started":1}
W20260621 02:45:24.579613 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:24.571346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:60707 (local address 127.9.166.62:34841)
0621 02:45:24.571587 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:24.571594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:24.571974 (+ 380us) server_negotiation.cc:407] Connection header received
0621 02:45:24.572163 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:24.572169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:24.572292 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:24.572391 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:24.573131 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:24.573875 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:24.575091 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:24.575591 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:24.575711 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:24.579332 (+ 3621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.1:60707: BlockingRecv error: recv got EOF from 127.9.166.1:60707 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Jun 21 02:45:25 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:25 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:25.031296 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:25.022199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.2:36817)
0621 02:45:25.022697 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:25.022713 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:25.022831 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:25.023135 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:25.023138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:25.023589 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:25.023754 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:25.023764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:25.025300 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:25.025305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:25.026412 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:25.026425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:25.027185 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:25.027190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:25.027326 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:25.028022 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:25.028040 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:25.030996 (+ 2956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":300,"threads_started":1}
W20260621 02:45:25.031647 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:25.022303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:36817 (local address 127.9.166.62:34841)
0621 02:45:25.022572 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:25.022578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:25.022797 (+ 219us) server_negotiation.cc:407] Connection header received
0621 02:45:25.022936 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:25.022940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:25.023004 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:25.023093 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:25.023925 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:25.025150 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:25.026552 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:25.027051 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:25.027153 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:25.031178 (+ 4025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.2:36817: BlockingRecv error: recv got EOF from 127.9.166.2:36817 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Jun 21 02:45:25 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:25 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:25.456633 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:25.448075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.3:34819)
0621 02:45:25.448779 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:25.448801 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:25.448938 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:25.449334 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:25.449337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:25.449802 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:25.450033 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:25.450047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:25.451262 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:25.451266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:25.452247 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:25.452258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:25.452965 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:25.452970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:25.453093 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:25.453706 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:25.453723 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:25.456403 (+ 2680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":479,"threads_started":1}
W20260621 02:45:25.456902 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:25.448276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:34819 (local address 127.9.166.62:34841)
0621 02:45:25.448532 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:25.448540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:25.448892 (+ 352us) server_negotiation.cc:407] Connection header received
0621 02:45:25.449073 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:25.449081 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:25.449190 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:25.449306 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:25.450196 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:25.451116 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:25.452379 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:25.452819 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:25.452937 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:25.456617 (+ 3680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.3:34819: BlockingRecv error: recv got EOF from 127.9.166.3:34819 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Jun 21 02:45:25 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:25 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:25.589119 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:25.580428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.1:48785)
0621 02:45:25.581118 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:25.581143 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:25.581305 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:25.581665 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:25.581670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:25.582120 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:25.582288 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:25.582296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:25.583329 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:25.583335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:25.584305 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:25.584317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:25.585076 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:25.585082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:25.585260 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:25.585817 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:25.585834 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:25.588773 (+ 2939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":479,"threads_started":1}
W20260621 02:45:25.589265 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:25.580497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:48785 (local address 127.9.166.62:34841)
0621 02:45:25.580745 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:25.580755 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:25.581237 (+ 482us) server_negotiation.cc:407] Connection header received
0621 02:45:25.581442 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:25.581448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:25.581529 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:25.581632 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:25.582436 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:25.583176 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:25.584459 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:25.584912 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:25.585056 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:25.589026 (+ 3970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.1:48785: BlockingRecv error: recv got EOF from 127.9.166.1:48785 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 21 02:45:26 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:26 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:26.041077 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:26.032310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.2:39545)
0621 02:45:26.032826 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:26.032841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:26.033008 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:26.033401 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:26.033406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:26.033836 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:26.034005 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:26.034014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:26.035060 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:26.035066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:26.036169 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:26.036182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:26.037021 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:26.037026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:26.037209 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:26.037858 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:26.037876 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:26.040845 (+ 2969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":279,"threads_started":1}
W20260621 02:45:26.041251 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:26.032381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:39545 (local address 127.9.166.62:34841)
0621 02:45:26.032641 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:26.032649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:26.032934 (+ 285us) server_negotiation.cc:407] Connection header received
0621 02:45:26.033188 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:26.033194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:26.033264 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:26.033369 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:26.034158 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:26.034909 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:26.036317 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:26.036857 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:26.036997 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:26.041029 (+ 4032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.2:39545: BlockingRecv error: recv got EOF from 127.9.166.2:39545 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 21 02:45:26 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:26 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:26.465564 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:26.457502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.3:41217)
0621 02:45:26.457956 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:26.457971 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:26.458077 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:26.458354 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:26.458357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:26.458738 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:26.458929 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:26.458937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:26.459954 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:26.459958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:26.460837 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:26.460847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:26.461585 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:26.461593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:26.461822 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:26.462571 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:26.462595 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:26.465328 (+ 2733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":282,"threads_started":1}
W20260621 02:45:26.465713 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:26.457541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:41217 (local address 127.9.166.62:34841)
0621 02:45:26.457758 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:26.457768 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:26.458050 (+ 282us) server_negotiation.cc:407] Connection header received
0621 02:45:26.458170 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:26.458174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:26.458234 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:26.458328 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:26.459051 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:26.459848 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:26.460951 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:26.461428 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:26.461547 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:26.465511 (+ 3964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.3:41217: BlockingRecv error: recv got EOF from 127.9.166.3:41217 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 21 02:45:26 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:26 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:26.598394 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:26.590125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.1:42629)
0621 02:45:26.590695 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:26.590717 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:26.590856 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:26.591151 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:26.591154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:26.591534 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:26.591675 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:26.591683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:26.592823 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:26.592829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:26.593962 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:26.593975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:26.594708 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:26.594712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:26.594921 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:26.595531 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:26.595557 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:26.598152 (+ 2595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":352,"threads_started":1}
W20260621 02:45:26.598563 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:26.590196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:42629 (local address 127.9.166.62:34841)
0621 02:45:26.590368 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:26.590372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:26.590801 (+ 429us) server_negotiation.cc:407] Connection header received
0621 02:45:26.590952 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:26.590956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:26.591028 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:26.591109 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:26.591806 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:26.592622 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:26.594107 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:26.594589 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:26.594682 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:26.598338 (+ 3656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.1:42629: BlockingRecv error: recv got EOF from 127.9.166.1:42629 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 21 02:45:27 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:27 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:27.053261 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:27.041965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.2:33913)
0621 02:45:27.042516 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:27.042532 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:27.042642 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:27.042929 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:27.042932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:27.043315 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:27.043565 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:27.043578 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:27.044928 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:27.044935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:27.045956 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:27.045968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:27.046696 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:27.046701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:27.046894 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:27.050177 (+ 3283us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:27.050195 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:27.053029 (+ 2834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":313,"threads_started":1}
W20260621 02:45:27.053483 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:27.042077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:33913 (local address 127.9.166.62:34841)
0621 02:45:27.042255 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:27.042264 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:27.042604 (+ 340us) server_negotiation.cc:407] Connection header received
0621 02:45:27.042734 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:27.042738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:27.042796 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:27.042876 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:27.043774 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:27.044776 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:27.046099 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:27.046546 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:27.046673 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:27.053234 (+ 6561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.2:33913: BlockingRecv error: recv got EOF from 127.9.166.2:33913 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 21 02:45:27 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:27 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:27.475271 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:27.466553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.3:54995)
0621 02:45:27.467109 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:27.467125 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:27.467246 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:27.467656 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:27.467664 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:27.468097 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:27.468272 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:27.468280 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:27.469631 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:27.469635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:27.470653 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:27.470664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:27.471516 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:27.471520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:27.471666 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:27.472229 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:27.472247 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:27.475040 (+ 2793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":312,"threads_started":1}
W20260621 02:45:27.475558 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:27.466601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:54995 (local address 127.9.166.62:34841)
0621 02:45:27.466766 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:27.466770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:27.467227 (+ 457us) server_negotiation.cc:407] Connection header received
0621 02:45:27.467370 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:27.467375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:27.467477 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:27.467587 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:27.468426 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:27.469458 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:27.470782 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:27.471343 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:27.471512 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:27.475274 (+ 3762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.3:54995: BlockingRecv error: recv got EOF from 127.9.166.3:54995 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 21 02:45:27 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:27 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:27.607335 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:27.599458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.1:35399)
0621 02:45:27.599921 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:27.599940 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:27.600049 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:27.600332 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:27.600335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:27.600701 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:27.600850 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:27.600858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:27.601836 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:27.601839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:27.602734 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:27.602745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:27.603479 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:27.603483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:27.603606 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:27.604149 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:27.604165 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:27.606983 (+ 2818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":248,"threads_started":1}
W20260621 02:45:27.607414 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:27.599542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:35399 (local address 127.9.166.62:34841)
0621 02:45:27.599768 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:27.599775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:27.600019 (+ 244us) server_negotiation.cc:407] Connection header received
0621 02:45:27.600149 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:27.600153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:27.600231 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:27.600308 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:27.600975 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:27.601721 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:27.602868 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:27.603316 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:27.603468 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:27.607164 (+ 3696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.1:35399: BlockingRecv error: recv got EOF from 127.9.166.1:35399 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 21 02:45:28 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:28 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:28.062265 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:28.054252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.2:58197)
0621 02:45:28.054786 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:28.054802 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:28.054910 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:28.055190 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:28.055193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:28.055579 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:28.055761 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:28.055771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:28.056795 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:28.056798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:28.057701 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:28.057718 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:28.058465 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:28.058469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:28.058575 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:28.059227 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:28.059244 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:28.062028 (+ 2784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":280,"threads_started":1}
W20260621 02:45:28.062459 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:28.054426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:58197 (local address 127.9.166.62:34841)
0621 02:45:28.054599 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:28.054603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:28.054875 (+ 272us) server_negotiation.cc:407] Connection header received
0621 02:45:28.055007 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:28.055011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:28.055072 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:28.055153 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:28.055920 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:28.056658 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:28.057847 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:28.058312 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:28.058405 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:28.062236 (+ 3831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.2:58197: BlockingRecv error: recv got EOF from 127.9.166.2:58197 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 21 02:45:28 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:28 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:28.484609 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:28.476298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.3:36783)
0621 02:45:28.476979 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:28.477000 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:28.477136 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:28.477482 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:28.477486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:28.477896 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:28.478050 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:28.478059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:28.479081 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:28.479088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:28.480020 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:28.480030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:28.480741 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:28.480747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:28.480928 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:28.481481 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:28.481498 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:28.484278 (+ 2780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":400,"threads_started":1}
W20260621 02:45:28.484717 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:28.476379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:36783 (local address 127.9.166.62:34841)
0621 02:45:28.476566 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:28.476573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:28.477096 (+ 523us) server_negotiation.cc:407] Connection header received
0621 02:45:28.477245 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:28.477252 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:28.477336 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:28.477449 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:28.478203 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:28.478947 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:28.480166 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:28.480565 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:28.480696 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:28.484475 (+ 3779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.3:36783: BlockingRecv error: recv got EOF from 127.9.166.3:36783 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 21 02:45:28 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:28 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:28.616467 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:28.608514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.1:50629)
0621 02:45:28.608975 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:28.608994 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:28.609108 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:28.609439 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:28.609442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:28.609795 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:28.609958 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:28.609968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:28.611067 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:28.611071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:28.612123 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:28.612136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:28.612842 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:28.612846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:28.612963 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:28.613555 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:28.613572 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:28.616241 (+ 2669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":243,"threads_started":1}
W20260621 02:45:28.616681 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:28.608553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:50629 (local address 127.9.166.62:34841)
0621 02:45:28.608849 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:28.608855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:28.609070 (+ 215us) server_negotiation.cc:407] Connection header received
0621 02:45:28.609203 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:28.609207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:28.609317 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:28.609404 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:28.610088 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:28.610920 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:28.612273 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:28.612707 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:28.612819 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:28.616429 (+ 3610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.1:50629: BlockingRecv error: recv got EOF from 127.9.166.1:50629 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
Jun 21 02:45:29 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:29 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:29.071722 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:29.063204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.2:49037)
0621 02:45:29.063748 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:29.063766 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:29.063876 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:29.064166 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:29.064170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:29.064540 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:29.064741 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:29.064754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:29.065762 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:29.065766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:29.066694 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:29.066705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:29.067434 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:29.067440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:29.067613 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:29.068325 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:29.068356 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:29.071474 (+ 3118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":327,"threads_started":1}
W20260621 02:45:29.071861 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:29.063328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:49037 (local address 127.9.166.62:34841)
0621 02:45:29.063521 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:29.063526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:29.063835 (+ 309us) server_negotiation.cc:407] Connection header received
0621 02:45:29.063968 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:29.063972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:29.064035 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:29.064115 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:29.064900 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:29.065628 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:29.066811 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:29.067263 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:29.067351 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:29.071651 (+ 4300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.2:49037: BlockingRecv error: recv got EOF from 127.9.166.2:49037 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 21 02:45:29 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:29 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.3@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:29.493643 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:29.485585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.3:38207)
0621 02:45:29.486007 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:29.486054 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:29.486165 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:29.486439 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:29.486442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:29.486782 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:29.486949 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:29.486956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:29.487943 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:29.487946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:29.488864 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:29.488875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:29.489494 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:29.489498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:29.489631 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:29.490186 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:29.490202 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:29.493280 (+ 3078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":238,"threads_started":1}
W20260621 02:45:29.493774 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:29.485656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:38207 (local address 127.9.166.62:34841)
0621 02:45:29.485891 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:29.485900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:29.486129 (+ 229us) server_negotiation.cc:407] Connection header received
0621 02:45:29.486255 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:29.486259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:29.486320 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:29.486398 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:29.487060 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:29.487821 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:29.488979 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:29.489374 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:29.489462 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:29.493499 (+ 4037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.3:38207: BlockingRecv error: recv got EOF from 127.9.166.3:38207 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 21 02:45:29 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:29 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.1@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:29.626641 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:29.617583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.1:59033)
0621 02:45:29.618057 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:29.618081 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:29.618201 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:29.618516 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:29.618519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:29.618926 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:29.619114 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:29.619123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:29.620324 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:29.620329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:29.621851 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:29.621862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:29.622767 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:29.622773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:29.622949 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:29.623576 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:29.623595 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:29.626290 (+ 2695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":272,"threads_started":1}
W20260621 02:45:29.626726 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:29.617639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:59033 (local address 127.9.166.62:34841)
0621 02:45:29.617853 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:29.617862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:29.618170 (+ 308us) server_negotiation.cc:407] Connection header received
0621 02:45:29.618303 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:29.618306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:29.618397 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:29.618478 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:29.619246 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:29.620148 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:29.621990 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:29.622613 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:29.622748 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:29.626480 (+ 3732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.1:59033: BlockingRecv error: recv got EOF from 127.9.166.1:59033 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 21 02:45:30 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
Jun 21 02:45:30 dist-test-slave-hvgn krb5kdc[18117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.166.2@KRBTEST.COM for oryx/127.9.166.62@KRBTEST.COM, Server not found in Kerberos database
I20260621 02:45:30.080803 19162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:30.072788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34841 (local address 127.9.166.2:50249)
0621 02:45:30.073332 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:30.073348 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:30.073481 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:30.073894 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:30.073899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:30.074373 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:30.074566 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:30.074575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:30.075625 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:30.075629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:30.076531 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:30.076541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:30.077217 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:30.077222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:30.077346 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:30.077902 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:30.077920 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:30.080545 (+ 2625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.166.62:34841: Server oryx/127.9.166.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":317,"threads_started":1}
W20260621 02:45:30.080950 18853 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:30.072876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:50249 (local address 127.9.166.62:34841)
0621 02:45:30.073074 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:30.073081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:30.073438 (+ 357us) server_negotiation.cc:407] Connection header received
0621 02:45:30.073620 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:30.073625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:30.073700 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:30.073826 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:30.074702 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:30.075487 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:30.076646 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:30.077068 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:30.077158 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:30.080739 (+ 3581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.166.2:50249: BlockingRecv error: recv got EOF from 127.9.166.2:50249 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:45:30.342000 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 18704
I20260621 02:45:30.354000 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:30.343055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37920 (local address 127.9.166.1:44419)
0621 02:45:30.343636 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:30.343644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:30.343669 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:45:30.343728 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:30.343734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:30.344177 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:30.344402 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:30.345759 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:30.346829 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:30.348059 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:30.348648 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:30.348750 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:30.350980 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:30.350988 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:30.351000 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:30.351029 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:30.353546 (+ 2517us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.166.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0621 02:45:30.353776 (+ 230us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37920: Request ticket server kudu/127.9.166.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":280,"threads_started":1}
W20260621 02:45:30.354115 19164 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37920: Request ticket server kudu/127.9.166.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260621 02:45:30.354136 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 18704: Not authorized: Client connection negotiation failed: client connection to 127.9.166.1:44419: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.166.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260621 02:45:30.354357 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 18704
I20260621 02:45:30.371806 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 18855
I20260621 02:45:30.383333 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:30.372788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33262 (local address 127.9.166.2:32919)
0621 02:45:30.373359 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:30.373366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:30.373396 (+ 30us) server_negotiation.cc:407] Connection header received
0621 02:45:30.373452 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:30.373459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:30.373671 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:30.373833 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:30.375040 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:30.376155 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:30.377333 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:30.377846 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:30.377940 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:30.380344 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:30.380353 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:30.380365 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:30.380397 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:30.382821 (+ 2424us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.166.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0621 02:45:30.383049 (+ 228us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33262: Request ticket server kudu/127.9.166.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":253,"threads_started":1}
W20260621 02:45:30.383450 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 18855: Not authorized: Client connection negotiation failed: client connection to 127.9.166.2:32919: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.166.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260621 02:45:30.383544 19165 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33262: Request ticket server kudu/127.9.166.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260621 02:45:30.383638 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 18855
I20260621 02:45:30.399561 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 18986
I20260621 02:45:30.410552 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:30.400416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54394 (local address 127.9.166.3:43313)
0621 02:45:30.400916 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:30.400923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:30.400947 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:45:30.401010 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:30.401017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:30.401226 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:30.401389 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:30.402563 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:30.403687 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:30.404786 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:30.405317 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:30.405412 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:30.407692 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:30.407699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:30.407712 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:30.407743 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:30.410104 (+ 2361us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.166.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0621 02:45:30.410334 (+ 230us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54394: Request ticket server kudu/127.9.166.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":267,"threads_started":1}
W20260621 02:45:30.410668 19166 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54394: Request ticket server kudu/127.9.166.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260621 02:45:30.410673 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 18986: Not authorized: Client connection negotiation failed: client connection to 127.9.166.3:43313: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.166.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260621 02:45:30.410832 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 18986
I20260621 02:45:30.426802 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 18635
I20260621 02:45:30.427974 18697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:30.543818 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 18635
2026-06-21T02:45:30Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20889 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-21T02:45:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:45:30Z Disabled control of system clock
I20260621 02:45:30.617668 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:36149
--webserver_interface=127.9.166.62
--webserver_port=0
--builtin_ntp_servers=127.9.166.20:38251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:36149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260621 02:45:30.936326 19171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:30.936606 19171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:30.936682 19171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:30.943619 19171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:45:30.943723 19171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:30.943759 19171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:30.943815 19171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:45:30.943859 19171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:45:30.952875 19171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:38251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:36149
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:36149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:30.954128 19171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:30.955976 19171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:30.967901 19176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:30.968000 19179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:30.967962 19177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:30.968673 19171 server_base.cc:1034] running on GCE node
I20260621 02:45:30.969508 19171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:30.970911 19171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:30.972148 19171 hybrid_clock.cc:648] HybridClock initialized: now 1782009930972100 us; error 58 us; skew 500 ppm
I20260621 02:45:30.972505 19171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:45:30.974687 19171 webserver.cc:460] Webserver started at http://127.9.166.62:42949/ using document root <none> and password file <none>
I20260621 02:45:30.975376 19171 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:30.975564 19171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:30.975893 19171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:45:30.978678 19171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "d7f77f05fbab404aac9beababd1ac47a"
format_stamp: "Formatted at 2026-06-21 02:45:30 on dist-test-slave-hvgn"
server_key: "812d5db8369d435f9cb14367ff25d163"
server_key_iv: "f164cbe48e8405fff356d643563ebcfb"
server_key_version: "encryptionkey@0"
I20260621 02:45:30.979503 19171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "d7f77f05fbab404aac9beababd1ac47a"
format_stamp: "Formatted at 2026-06-21 02:45:30 on dist-test-slave-hvgn"
server_key: "812d5db8369d435f9cb14367ff25d163"
server_key_iv: "f164cbe48e8405fff356d643563ebcfb"
server_key_version: "encryptionkey@0"
I20260621 02:45:30.984395 19171 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 02:45:30.987718 19185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:30.988916 19171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260621 02:45:30.989130 19171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "d7f77f05fbab404aac9beababd1ac47a"
format_stamp: "Formatted at 2026-06-21 02:45:30 on dist-test-slave-hvgn"
server_key: "812d5db8369d435f9cb14367ff25d163"
server_key_iv: "f164cbe48e8405fff356d643563ebcfb"
server_key_version: "encryptionkey@0"
I20260621 02:45:30.989331 19171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:31.006135 19171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:31.006971 19171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:31.007256 19171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:31.026260 19171 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:36149
I20260621 02:45:31.026278 19236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:36149 every 8 connection(s)
I20260621 02:45:31.027686 19171 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:45:31.031278 19237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:31.036654 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 19171
I20260621 02:45:31.036870 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:45:31.037679 9880 external_mini_cluster.cc:1428] Setting key ab0777921cb76975b69b694dd50ffb49
I20260621 02:45:31.043015 19237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a: Bootstrap starting.
I20260621 02:45:31.046518 19237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:31.047700 19237 log.cc:824] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a: Log is configured to *not* fsync() on all Append() calls
I20260621 02:45:31.050982 19237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a: No bootstrap required, opened a new log
I20260621 02:45:31.052119 19240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:31.039741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54278 (local address 127.9.166.62:36149)
0621 02:45:31.040760 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:31.040770 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:31.040811 (+ 41us) server_negotiation.cc:407] Connection header received
0621 02:45:31.042389 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:31.042409 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:31.042856 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:31.043226 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:31.043834 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:31.045109 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:31.047037 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:31.047914 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:31.048052 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:31.049349 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:31.049359 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:31.049375 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:45:31.050055 (+ 680us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:31.050166 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:45:31.050192 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:31.050311 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:31.050525 (+ 214us) server_negotiation.cc:299] Negotiation successful
0621 02:45:31.050849 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":393,"threads_started":1}
I20260621 02:45:31.059018 19237 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f77f05fbab404aac9beababd1ac47a" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36149 } }
I20260621 02:45:31.059334 19237 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:31.059461 19237 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f77f05fbab404aac9beababd1ac47a, State: Initialized, Role: FOLLOWER
I20260621 02:45:31.059928 19237 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f77f05fbab404aac9beababd1ac47a" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36149 } }
I20260621 02:45:31.060110 19237 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:45:31.060205 19237 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:45:31.060365 19237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:31.062649 19237 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f77f05fbab404aac9beababd1ac47a" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36149 } }
I20260621 02:45:31.063124 19237 leader_election.cc:302] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7f77f05fbab404aac9beababd1ac47a; no voters:
I20260621 02:45:31.063724 19237 leader_election.cc:288] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:45:31.063861 19242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:45:31.064544 19242 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a [term 1 LEADER]: Becoming Leader. State: Replica: d7f77f05fbab404aac9beababd1ac47a, State: Running, Role: LEADER
I20260621 02:45:31.065023 19242 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f77f05fbab404aac9beababd1ac47a" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36149 } }
I20260621 02:45:31.065696 19237 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:45:31.067595 19243 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7f77f05fbab404aac9beababd1ac47a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f77f05fbab404aac9beababd1ac47a" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36149 } } }
I20260621 02:45:31.067966 19243 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a [sys.catalog]: This master's current role is: LEADER
I20260621 02:45:31.068358 19244 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7f77f05fbab404aac9beababd1ac47a. Latest consensus state: current_term: 1 leader_uuid: "d7f77f05fbab404aac9beababd1ac47a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7f77f05fbab404aac9beababd1ac47a" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36149 } } }
I20260621 02:45:31.068732 19244 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a [sys.catalog]: This master's current role is: LEADER
W20260621 02:45:31.078459 19258 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:45:31.078627 19258 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:45:31.079169 19249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:45:31.081215 19249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:45:31.089284 19249 catalog_manager.cc:1347] Generated new cluster ID: 54784c833e8c4041a5b17726b9257b4e
I20260621 02:45:31.089393 19249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:45:31.107185 19249 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:45:31.108525 19249 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:45:31.119571 19249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7f77f05fbab404aac9beababd1ac47a: Generated new TSK 0
I20260621 02:45:31.120358 19249 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:45:31.125447 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:36149
--builtin_ntp_servers=127.9.166.20:38251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260621 02:45:31.475835 19261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:31.476133 19261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:31.476292 19261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:31.483316 19261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:31.483474 19261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:31.483649 19261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:45:31.492885 19261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:38251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:36149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:31.494163 19261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:31.495947 19261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:31.508591 19267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:31.508525 19269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:31.508772 19266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:31.509233 19261 server_base.cc:1034] running on GCE node
I20260621 02:45:31.509842 19261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:31.510794 19261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:31.512017 19261 hybrid_clock.cc:648] HybridClock initialized: now 1782009931511955 us; error 56 us; skew 500 ppm
I20260621 02:45:31.512326 19261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:45:31.514518 19261 webserver.cc:460] Webserver started at http://127.9.166.1:38589/ using document root <none> and password file <none>
I20260621 02:45:31.515194 19261 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:31.515326 19261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:31.515663 19261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:45:31.518433 19261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "26d30bd4bf864f8cb606dad069fb8b24"
format_stamp: "Formatted at 2026-06-21 02:45:31 on dist-test-slave-hvgn"
server_key: "a2f254618fced8737b2c0c8cc399fb4e"
server_key_iv: "ebe8553bf06c1dba658da95efc480109"
server_key_version: "encryptionkey@0"
I20260621 02:45:31.519198 19261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "26d30bd4bf864f8cb606dad069fb8b24"
format_stamp: "Formatted at 2026-06-21 02:45:31 on dist-test-slave-hvgn"
server_key: "a2f254618fced8737b2c0c8cc399fb4e"
server_key_iv: "ebe8553bf06c1dba658da95efc480109"
server_key_version: "encryptionkey@0"
I20260621 02:45:31.523970 19261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 02:45:31.527141 19275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:31.528379 19261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 02:45:31.528549 19261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "26d30bd4bf864f8cb606dad069fb8b24"
format_stamp: "Formatted at 2026-06-21 02:45:31 on dist-test-slave-hvgn"
server_key: "a2f254618fced8737b2c0c8cc399fb4e"
server_key_iv: "ebe8553bf06c1dba658da95efc480109"
server_key_version: "encryptionkey@0"
I20260621 02:45:31.528750 19261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:31.555980 19261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:31.556886 19261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:31.557175 19261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:31.558288 19261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:45:31.559859 19261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:45:31.560109 19261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:31.560269 19261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:45:31.560369 19261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:31.593091 19261 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:42029
I20260621 02:45:31.593175 19387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:42029 every 8 connection(s)
I20260621 02:45:31.594404 19261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:45:31.600541 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 19261
I20260621 02:45:31.600744 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:45:31.601411 9880 external_mini_cluster.cc:1428] Setting key 88d87e4ba5e4f259510626a6e9b3d164
I20260621 02:45:31.603983 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:36149
--builtin_ntp_servers=127.9.166.20:38251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260621 02:45:31.607211 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:31.596949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:50385 (local address 127.9.166.62:36149)
0621 02:45:31.597385 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:31.597390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:31.598598 (+ 1208us) server_negotiation.cc:407] Connection header received
0621 02:45:31.600462 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:31.600468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:31.600597 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:31.600736 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:31.601984 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:31.602766 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:31.604468 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:31.604866 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:31.605011 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:31.606358 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:31.606363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:31.606366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:45:31.606384 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:31.606433 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:45:31.606444 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:31.606545 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:31.606782 (+ 237us) server_negotiation.cc:299] Negotiation successful
0621 02:45:31.606880 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":217,"threads_started":1}
I20260621 02:45:31.608264 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:31.597560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:36149 (local address 127.9.166.1:50385)
0621 02:45:31.598418 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:31.598463 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:31.600179 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:31.601039 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:31.601048 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:31.601140 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:31.601806 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:31.601824 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:31.602919 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:31.602929 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:31.604321 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:31.604333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:31.605091 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:31.605100 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:31.605335 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:31.606045 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:45:31.606048 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:31.606089 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:45:31.606170 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:45:31.606582 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:45:31.606591 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:45:31.606597 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 02:45:31.606777 (+ 180us) client_negotiation.cc:240] Negotiation successful
0621 02:45:31.607175 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":253,"threads_started":1}
I20260621 02:45:31.609776 19388 heartbeater.cc:347] Connected to a master server at 127.9.166.62:36149
I20260621 02:45:31.610090 19388 heartbeater.cc:464] Registering TS with master...
I20260621 02:45:31.610787 19388 heartbeater.cc:511] Master 127.9.166.62:36149 requested a full tablet report, sending...
I20260621 02:45:31.612774 19202 ts_manager.cc:194] Registered new tserver with Master: 26d30bd4bf864f8cb606dad069fb8b24 (127.9.166.1:42029)
I20260621 02:45:31.614264 19202 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.166.1:50385
W20260621 02:45:31.922554 19392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:31.922874 19392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:31.923019 19392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:31.930159 19392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:31.930290 19392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:31.930462 19392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:45:31.939635 19392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:38251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:36149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:31.940881 19392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:31.942613 19392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:31.955456 19397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:31.955420 19398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:31.955379 19400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:31.956666 19392 server_base.cc:1034] running on GCE node
I20260621 02:45:31.957247 19392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:31.958123 19392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:31.959379 19392 hybrid_clock.cc:648] HybridClock initialized: now 1782009931959338 us; error 55 us; skew 500 ppm
I20260621 02:45:31.959738 19392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:45:31.961882 19392 webserver.cc:460] Webserver started at http://127.9.166.2:39005/ using document root <none> and password file <none>
I20260621 02:45:31.962563 19392 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:31.962692 19392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:31.963006 19392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:45:31.965911 19392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "09ee16331b714eca976066f7bc553a8d"
format_stamp: "Formatted at 2026-06-21 02:45:31 on dist-test-slave-hvgn"
server_key: "50d5888536b648f2191e9b9e70eb29f0"
server_key_iv: "f27fc1a6da182ed2c7ae8f100b0f8061"
server_key_version: "encryptionkey@0"
I20260621 02:45:31.966701 19392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "09ee16331b714eca976066f7bc553a8d"
format_stamp: "Formatted at 2026-06-21 02:45:31 on dist-test-slave-hvgn"
server_key: "50d5888536b648f2191e9b9e70eb29f0"
server_key_iv: "f27fc1a6da182ed2c7ae8f100b0f8061"
server_key_version: "encryptionkey@0"
I20260621 02:45:31.971673 19392 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260621 02:45:31.974920 19406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:31.976313 19392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:45:31.976483 19392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "09ee16331b714eca976066f7bc553a8d"
format_stamp: "Formatted at 2026-06-21 02:45:31 on dist-test-slave-hvgn"
server_key: "50d5888536b648f2191e9b9e70eb29f0"
server_key_iv: "f27fc1a6da182ed2c7ae8f100b0f8061"
server_key_version: "encryptionkey@0"
I20260621 02:45:31.976684 19392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:32.000289 19392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:32.001119 19392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:32.001405 19392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:32.002346 19392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:45:32.003850 19392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:45:32.003973 19392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:32.004103 19392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:45:32.004197 19392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:32.036185 19392 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:33175
I20260621 02:45:32.036247 19518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:33175 every 8 connection(s)
I20260621 02:45:32.037460 19392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:45:32.045527 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 19392
I20260621 02:45:32.045706 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:45:32.046181 9880 external_mini_cluster.cc:1428] Setting key 7affa2af1c9c62d83334b1b45ac103da
I20260621 02:45:32.049125 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:36149
--builtin_ntp_servers=127.9.166.20:38251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260621 02:45:32.049976 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:32.040009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:59135 (local address 127.9.166.62:36149)
0621 02:45:32.040186 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:32.040191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:32.041715 (+ 1524us) server_negotiation.cc:407] Connection header received
0621 02:45:32.043474 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:32.043480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:32.043563 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:32.043668 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:32.044654 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:32.045362 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:32.047010 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:32.047669 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:32.047798 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:32.049287 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:32.049296 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:32.049303 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:45:32.049328 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:32.049368 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:45:32.049386 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:32.049454 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:32.049699 (+ 245us) server_negotiation.cc:299] Negotiation successful
0621 02:45:32.049771 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 02:45:32.051468 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:32.040648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:36149 (local address 127.9.166.2:59135)
0621 02:45:32.041563 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:32.041603 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:32.043169 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:32.043886 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:32.043894 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:32.043982 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:32.044499 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:32.044514 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:32.045511 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:32.045521 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:32.046866 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:32.046878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:32.047906 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:32.047913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:32.048193 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:32.048904 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:45:32.048908 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:32.048959 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:45:32.049059 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:45:32.049483 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:45:32.049496 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:45:32.049506 (+ 10us) client_negotiation.cc:769] Sending connection context
0621 02:45:32.049684 (+ 178us) client_negotiation.cc:240] Negotiation successful
0621 02:45:32.050102 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":290,"threads_started":1}
I20260621 02:45:32.052743 19519 heartbeater.cc:347] Connected to a master server at 127.9.166.62:36149
I20260621 02:45:32.053032 19519 heartbeater.cc:464] Registering TS with master...
I20260621 02:45:32.053754 19519 heartbeater.cc:511] Master 127.9.166.62:36149 requested a full tablet report, sending...
I20260621 02:45:32.055106 19202 ts_manager.cc:194] Registered new tserver with Master: 09ee16331b714eca976066f7bc553a8d (127.9.166.2:33175)
I20260621 02:45:32.055927 19202 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.166.2:59135
W20260621 02:45:32.365865 19522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:32.366158 19522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:32.366315 19522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:32.373524 19522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:32.373658 19522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:32.373829 19522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:45:32.382818 19522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:38251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:36149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:32.384095 19522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:32.385812 19522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:32.398197 19527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:32.398284 19530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:32.398248 19528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:32.399219 19522 server_base.cc:1034] running on GCE node
I20260621 02:45:32.400018 19522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:32.401072 19522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:32.402297 19522 hybrid_clock.cc:648] HybridClock initialized: now 1782009932402249 us; error 73 us; skew 500 ppm
I20260621 02:45:32.402607 19522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:45:32.409538 19522 webserver.cc:460] Webserver started at http://127.9.166.3:39391/ using document root <none> and password file <none>
I20260621 02:45:32.410243 19522 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:32.410372 19522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:32.410683 19522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:45:32.413417 19522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "c64f31cf29e1419abbb1411d78eae7be"
format_stamp: "Formatted at 2026-06-21 02:45:32 on dist-test-slave-hvgn"
server_key: "63173dd27baa6d6413a25d187aca9a78"
server_key_iv: "eaa313b5321d0c3914c58541ad9c56a1"
server_key_version: "encryptionkey@0"
I20260621 02:45:32.414161 19522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "c64f31cf29e1419abbb1411d78eae7be"
format_stamp: "Formatted at 2026-06-21 02:45:32 on dist-test-slave-hvgn"
server_key: "63173dd27baa6d6413a25d187aca9a78"
server_key_iv: "eaa313b5321d0c3914c58541ad9c56a1"
server_key_version: "encryptionkey@0"
I20260621 02:45:32.419023 19522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260621 02:45:32.422421 19536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:32.423803 19522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260621 02:45:32.423975 19522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "c64f31cf29e1419abbb1411d78eae7be"
format_stamp: "Formatted at 2026-06-21 02:45:32 on dist-test-slave-hvgn"
server_key: "63173dd27baa6d6413a25d187aca9a78"
server_key_iv: "eaa313b5321d0c3914c58541ad9c56a1"
server_key_version: "encryptionkey@0"
I20260621 02:45:32.424178 19522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:32.453815 19522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:32.454667 19522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:32.454936 19522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:32.455904 19522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:45:32.457453 19522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:45:32.457573 19522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:32.457708 19522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:45:32.457816 19522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:32.489285 19522 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:34201
I20260621 02:45:32.489408 19648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:34201 every 8 connection(s)
I20260621 02:45:32.490512 19522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:45:32.491431 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 19522
I20260621 02:45:32.491588 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:45:32.492020 9880 external_mini_cluster.cc:1428] Setting key 493d17f85180474e3988773250e0b052
I20260621 02:45:32.503010 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:32.493042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:40811 (local address 127.9.166.62:36149)
0621 02:45:32.493244 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:32.493249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:32.495828 (+ 2579us) server_negotiation.cc:407] Connection header received
0621 02:45:32.497426 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:32.497430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:32.497488 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:32.497566 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:32.498513 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:32.499210 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:32.500720 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:32.501130 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:32.501220 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:32.502373 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:32.502378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:32.502381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:45:32.502396 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:32.502426 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:45:32.502436 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:32.502503 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:32.502679 (+ 176us) server_negotiation.cc:299] Negotiation successful
0621 02:45:32.502749 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:45:32.503937 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:32.493666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:36149 (local address 127.9.166.3:40811)
0621 02:45:32.495677 (+ 2011us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:32.495725 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:32.497205 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:32.497758 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:32.497765 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:32.497847 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:32.498387 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:32.498402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:32.499322 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:32.499325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:32.500604 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:32.500615 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:32.501312 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:32.501316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:32.501500 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:32.502096 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:45:32.502098 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:32.502132 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:45:32.502198 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:45:32.502516 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:45:32.502523 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:45:32.502528 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 02:45:32.502654 (+ 126us) client_negotiation.cc:240] Negotiation successful
0621 02:45:32.502899 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1527,"thread_start_us":1315,"threads_started":1}
I20260621 02:45:32.505165 19649 heartbeater.cc:347] Connected to a master server at 127.9.166.62:36149
I20260621 02:45:32.505447 19649 heartbeater.cc:464] Registering TS with master...
I20260621 02:45:32.506173 19649 heartbeater.cc:511] Master 127.9.166.62:36149 requested a full tablet report, sending...
I20260621 02:45:32.507550 19202 ts_manager.cc:194] Registered new tserver with Master: c64f31cf29e1419abbb1411d78eae7be (127.9.166.3:34201)
I20260621 02:45:32.508301 19202 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.166.3:40811
I20260621 02:45:32.518807 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 02:45:32.524633 19390 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:32.523187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50618 (local address 127.9.166.62:36149)
0621 02:45:32.523439 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:32.523448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:32.523814 (+ 366us) server_negotiation.cc:407] Connection header received
0621 02:45:32.523930 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:32.523933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:32.524001 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:32.524076 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:32.524429 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50618: BlockingRecv error: recv got EOF from 127.0.0.1:50618 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:45:32.525007 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 19261
I20260621 02:45:32.532387 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:32.525912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34240 (local address 127.9.166.1:42029)
0621 02:45:32.526441 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:32.526455 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:32.526479 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:45:32.526529 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:32.526534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:32.526715 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:32.526870 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:32.527210 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:32.528401 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:32.529554 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:32.530072 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:32.530163 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:32.531114 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:32.531121 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:32.531133 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:45:32.531733 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:32.531778 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:45:32.531795 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:32.531890 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:32.532012 (+ 122us) server_negotiation.cc:299] Negotiation successful
0621 02:45:32.532109 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":288,"threads_started":1}
I20260621 02:45:32.533465 19383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:32.615032 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 19261
I20260621 02:45:32.628365 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 19392
I20260621 02:45:32.635699 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:32.629252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47220 (local address 127.9.166.2:33175)
0621 02:45:32.629739 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:32.629746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:32.629771 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:45:32.629849 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:32.629855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:32.630067 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:32.630193 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:32.630547 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:32.631590 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:32.632721 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:32.633229 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:32.633327 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:32.634199 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:32.634206 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:32.634219 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:45:32.634829 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:32.634874 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:45:32.634894 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:32.634982 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:32.635153 (+ 171us) server_negotiation.cc:299] Negotiation successful
0621 02:45:32.635313 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":256,"threads_started":1}
I20260621 02:45:32.636953 19514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:32.717463 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 19392
I20260621 02:45:32.733280 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 19522
I20260621 02:45:32.741511 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:32.734216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.9.166.3:34201)
0621 02:45:32.734878 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:32.734885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:32.734917 (+ 32us) server_negotiation.cc:407] Connection header received
0621 02:45:32.734979 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:32.734985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:32.735211 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:32.735353 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:32.735872 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:32.736966 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:32.738305 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:32.738884 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:32.738989 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:32.740084 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:32.740092 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:32.740105 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:45:32.740692 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:32.740746 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:45:32.740765 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:32.740861 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:32.740992 (+ 131us) server_negotiation.cc:299] Negotiation successful
0621 02:45:32.741151 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":315,"threads_started":1}
I20260621 02:45:32.742702 19644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:32.823931 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 19522
I20260621 02:45:32.840387 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 19171
I20260621 02:45:32.841492 19232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:32.942410 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 19171
2026-06-21T02:45:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2393 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-21T02:45:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:45:32Z Disabled control of system clock
I20260621 02:45:33.011051 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:40615
--webserver_interface=127.9.166.62
--webserver_port=0
--builtin_ntp_servers=127.9.166.20:38921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:40615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 02:45:33.325727 19668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:33.326031 19668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:33.326115 19668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:33.333236 19668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:45:33.333324 19668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:33.333362 19668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:33.333416 19668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:45:33.333459 19668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:45:33.342383 19668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:38921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:40615
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:40615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:33.343564 19668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:33.345196 19668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:33.359679 19673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:33.359756 19674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:33.360211 19676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:33.360702 19668 server_base.cc:1034] running on GCE node
I20260621 02:45:33.361426 19668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:33.362702 19668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:33.363919 19668 hybrid_clock.cc:648] HybridClock initialized: now 1782009933363869 us; error 62 us; skew 500 ppm
I20260621 02:45:33.364245 19668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:45:33.366391 19668 webserver.cc:460] Webserver started at http://127.9.166.62:37025/ using document root <none> and password file <none>
I20260621 02:45:33.367061 19668 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:33.367188 19668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:33.367568 19668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:45:33.370348 19668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "39cb8c36d8b44d39b4e512db5494f784"
format_stamp: "Formatted at 2026-06-21 02:45:33 on dist-test-slave-hvgn"
server_key: "02b71b2bb3d0685c11b310e267fcbfdb"
server_key_iv: "3e0d147ec0c1032c7c83535870ce7a77"
server_key_version: "encryptionkey@0"
I20260621 02:45:33.371122 19668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "39cb8c36d8b44d39b4e512db5494f784"
format_stamp: "Formatted at 2026-06-21 02:45:33 on dist-test-slave-hvgn"
server_key: "02b71b2bb3d0685c11b310e267fcbfdb"
server_key_iv: "3e0d147ec0c1032c7c83535870ce7a77"
server_key_version: "encryptionkey@0"
I20260621 02:45:33.375919 19668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260621 02:45:33.379020 19682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:33.380252 19668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260621 02:45:33.380438 19668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "39cb8c36d8b44d39b4e512db5494f784"
format_stamp: "Formatted at 2026-06-21 02:45:33 on dist-test-slave-hvgn"
server_key: "02b71b2bb3d0685c11b310e267fcbfdb"
server_key_iv: "3e0d147ec0c1032c7c83535870ce7a77"
server_key_version: "encryptionkey@0"
I20260621 02:45:33.380643 19668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:33.386071 19668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:33.386788 19668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:33.387121 19668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:33.404831 19668 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:40615
I20260621 02:45:33.404865 19733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:40615 every 8 connection(s)
I20260621 02:45:33.406136 19668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:45:33.409663 19734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:33.409966 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 19668
I20260621 02:45:33.410172 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:45:33.410792 9880 external_mini_cluster.cc:1428] Setting key 289d310199fa42763b993ac84dd695f1
I20260621 02:45:33.419734 19737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:33.412953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39176 (local address 127.9.166.62:40615)
0621 02:45:33.414171 (+ 1218us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:33.414193 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:33.414347 (+ 154us) server_negotiation.cc:407] Connection header received
0621 02:45:33.416016 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:33.416042 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:33.416563 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:33.416993 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:33.417903 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:33.417911 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:33.417926 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:45:33.417941 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:33.418008 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:45:33.418028 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:33.418227 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:33.418347 (+ 120us) server_negotiation.cc:299] Negotiation successful
0621 02:45:33.418602 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":441,"threads_started":1}
I20260621 02:45:33.423624 19734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784: Bootstrap starting.
I20260621 02:45:33.426245 19734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:33.427210 19734 log.cc:824] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784: Log is configured to *not* fsync() on all Append() calls
I20260621 02:45:33.429713 19734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784: No bootstrap required, opened a new log
I20260621 02:45:33.436669 19734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39cb8c36d8b44d39b4e512db5494f784" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40615 } }
I20260621 02:45:33.436995 19734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:33.437106 19734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39cb8c36d8b44d39b4e512db5494f784, State: Initialized, Role: FOLLOWER
I20260621 02:45:33.437685 19734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39cb8c36d8b44d39b4e512db5494f784" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40615 } }
I20260621 02:45:33.437899 19734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:45:33.438033 19734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:45:33.438205 19734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:33.440514 19734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39cb8c36d8b44d39b4e512db5494f784" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40615 } }
I20260621 02:45:33.441062 19734 leader_election.cc:302] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39cb8c36d8b44d39b4e512db5494f784; no voters:
I20260621 02:45:33.441598 19734 leader_election.cc:288] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:45:33.441738 19739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:45:33.442502 19739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784 [term 1 LEADER]: Becoming Leader. State: Replica: 39cb8c36d8b44d39b4e512db5494f784, State: Running, Role: LEADER
I20260621 02:45:33.443027 19739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39cb8c36d8b44d39b4e512db5494f784" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40615 } }
I20260621 02:45:33.443689 19734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:45:33.445874 19741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39cb8c36d8b44d39b4e512db5494f784. Latest consensus state: current_term: 1 leader_uuid: "39cb8c36d8b44d39b4e512db5494f784" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39cb8c36d8b44d39b4e512db5494f784" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40615 } } }
I20260621 02:45:33.446071 19740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39cb8c36d8b44d39b4e512db5494f784" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39cb8c36d8b44d39b4e512db5494f784" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40615 } } }
I20260621 02:45:33.446316 19741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784 [sys.catalog]: This master's current role is: LEADER
I20260621 02:45:33.446389 19740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784 [sys.catalog]: This master's current role is: LEADER
I20260621 02:45:33.447764 19748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:45:33.454703 19748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:45:33.463362 19748 catalog_manager.cc:1347] Generated new cluster ID: 88a862a6cb7b49d29191dff4befb7229
I20260621 02:45:33.463526 19748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:45:33.479295 19748 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:45:33.479718 19748 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:45:33.485003 19748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39cb8c36d8b44d39b4e512db5494f784: Generated new TSK 0
I20260621 02:45:33.485847 19748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:45:33.492985 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:40615
--builtin_ntp_servers=127.9.166.20:38921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260621 02:45:33.811046 19758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:33.811318 19758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:33.811486 19758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:33.818420 19758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:33.818513 19758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:33.818616 19758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:45:33.827644 19758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:38921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:40615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:33.828838 19758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:33.830492 19758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:33.844501 19763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:33.844466 19764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:33.844429 19766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:33.845606 19758 server_base.cc:1034] running on GCE node
I20260621 02:45:33.846256 19758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:33.847173 19758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:33.848366 19758 hybrid_clock.cc:648] HybridClock initialized: now 1782009933848300 us; error 55 us; skew 500 ppm
I20260621 02:45:33.848707 19758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:45:33.850919 19758 webserver.cc:460] Webserver started at http://127.9.166.1:41439/ using document root <none> and password file <none>
I20260621 02:45:33.851666 19758 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:33.851758 19758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:33.852030 19758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:45:33.854820 19758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "b1443aecb96642e68314b187131b67a9"
format_stamp: "Formatted at 2026-06-21 02:45:33 on dist-test-slave-hvgn"
server_key: "d75c2c7eaddadf77ae380dc486a95661"
server_key_iv: "7dd7cf2fa2c0c1dbf01812fbf4668058"
server_key_version: "encryptionkey@0"
I20260621 02:45:33.855643 19758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "b1443aecb96642e68314b187131b67a9"
format_stamp: "Formatted at 2026-06-21 02:45:33 on dist-test-slave-hvgn"
server_key: "d75c2c7eaddadf77ae380dc486a95661"
server_key_iv: "7dd7cf2fa2c0c1dbf01812fbf4668058"
server_key_version: "encryptionkey@0"
I20260621 02:45:33.860571 19758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:45:33.863911 19772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:33.865252 19758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:45:33.865417 19758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "b1443aecb96642e68314b187131b67a9"
format_stamp: "Formatted at 2026-06-21 02:45:33 on dist-test-slave-hvgn"
server_key: "d75c2c7eaddadf77ae380dc486a95661"
server_key_iv: "7dd7cf2fa2c0c1dbf01812fbf4668058"
server_key_version: "encryptionkey@0"
I20260621 02:45:33.865628 19758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:33.871333 19758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:33.872226 19758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:33.872551 19758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:33.873525 19758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:45:33.875010 19758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:45:33.875141 19758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:33.875278 19758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:45:33.875366 19758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:33.909569 19758 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:38859
I20260621 02:45:33.909662 19884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:38859 every 8 connection(s)
I20260621 02:45:33.910908 19758 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:45:33.913712 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 19758
I20260621 02:45:33.913900 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:45:33.914490 9880 external_mini_cluster.cc:1428] Setting key fd76065487f0f55d841227eeac837c4b
I20260621 02:45:33.917090 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:40615
--builtin_ntp_servers=127.9.166.20:38921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260621 02:45:33.919504 19737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:33.913690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:41011 (local address 127.9.166.62:40615)
0621 02:45:33.913910 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:33.913916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:33.915535 (+ 1619us) server_negotiation.cc:407] Connection header received
0621 02:45:33.917213 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:33.917221 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:33.917289 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:33.917395 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:33.918738 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:33.918744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:33.918750 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:45:33.918771 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:33.918801 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:45:33.918812 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:33.918935 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:33.919135 (+ 200us) server_negotiation.cc:299] Negotiation successful
0621 02:45:33.919203 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:45:33.920552 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:33.914279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:40615 (local address 127.9.166.1:41011)
0621 02:45:33.915353 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:33.915427 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:33.916960 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:33.917601 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:33.917610 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:33.917700 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:33.918420 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:45:33.918423 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:33.918461 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:45:33.918551 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:45:33.918908 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:45:33.918918 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:45:33.918925 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 02:45:33.919219 (+ 294us) client_negotiation.cc:240] Negotiation successful
0621 02:45:33.919519 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":332,"threads_started":1}
I20260621 02:45:33.921995 19885 heartbeater.cc:347] Connected to a master server at 127.9.166.62:40615
I20260621 02:45:33.922290 19885 heartbeater.cc:464] Registering TS with master...
I20260621 02:45:33.922897 19885 heartbeater.cc:511] Master 127.9.166.62:40615 requested a full tablet report, sending...
I20260621 02:45:33.925268 19699 ts_manager.cc:194] Registered new tserver with Master: b1443aecb96642e68314b187131b67a9 (127.9.166.1:38859)
W20260621 02:45:34.237701 19888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:34.237958 19888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:34.238042 19888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:34.245342 19888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:34.245433 19888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:34.245538 19888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:45:34.254436 19888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:38921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:40615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:34.255658 19888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:34.257326 19888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:34.270095 19893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:34.270162 19896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:34.270179 19894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:34.270861 19888 server_base.cc:1034] running on GCE node
I20260621 02:45:34.271504 19888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:34.272398 19888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:34.273597 19888 hybrid_clock.cc:648] HybridClock initialized: now 1782009934273547 us; error 41 us; skew 500 ppm
I20260621 02:45:34.273934 19888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:45:34.276237 19888 webserver.cc:460] Webserver started at http://127.9.166.2:33699/ using document root <none> and password file <none>
I20260621 02:45:34.276934 19888 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:34.277046 19888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:34.277388 19888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:45:34.280226 19888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "37f8f99c442c4fe781f6321b46f6958e"
format_stamp: "Formatted at 2026-06-21 02:45:34 on dist-test-slave-hvgn"
server_key: "5ffdcf10cbbc0dd2f784da71881f4473"
server_key_iv: "bab646f3479669408d9227027721ff3d"
server_key_version: "encryptionkey@0"
I20260621 02:45:34.281009 19888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "37f8f99c442c4fe781f6321b46f6958e"
format_stamp: "Formatted at 2026-06-21 02:45:34 on dist-test-slave-hvgn"
server_key: "5ffdcf10cbbc0dd2f784da71881f4473"
server_key_iv: "bab646f3479669408d9227027721ff3d"
server_key_version: "encryptionkey@0"
I20260621 02:45:34.286078 19888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:45:34.289361 19902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:34.290678 19888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260621 02:45:34.290829 19888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "37f8f99c442c4fe781f6321b46f6958e"
format_stamp: "Formatted at 2026-06-21 02:45:34 on dist-test-slave-hvgn"
server_key: "5ffdcf10cbbc0dd2f784da71881f4473"
server_key_iv: "bab646f3479669408d9227027721ff3d"
server_key_version: "encryptionkey@0"
I20260621 02:45:34.291031 19888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:34.296496 19888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:34.297230 19888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:34.297482 19888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:34.298327 19888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:45:34.299764 19888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:45:34.299870 19888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:34.299994 19888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:45:34.300081 19888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:34.331969 19888 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:40831
I20260621 02:45:34.332019 20014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:40831 every 8 connection(s)
I20260621 02:45:34.333284 19888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:45:34.340857 19737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:34.335719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:46683 (local address 127.9.166.62:40615)
0621 02:45:34.335943 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:34.335949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:34.337275 (+ 1326us) server_negotiation.cc:407] Connection header received
0621 02:45:34.338888 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:34.338893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:34.338945 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:34.339035 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:34.340284 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:34.340288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:34.340291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:45:34.340306 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:34.340336 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:45:34.340347 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:34.340441 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:34.340597 (+ 156us) server_negotiation.cc:299] Negotiation successful
0621 02:45:34.340645 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260621 02:45:34.341180 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 19888
I20260621 02:45:34.341367 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:45:34.341899 9880 external_mini_cluster.cc:1428] Setting key 75d7e53ae19627f8ddaef05ba2356e59
I20260621 02:45:34.342056 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:34.336254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:40615 (local address 127.9.166.2:46683)
0621 02:45:34.337108 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:34.337151 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:34.338617 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:34.339227 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:34.339235 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:34.339322 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:34.340008 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:45:34.340011 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:34.340046 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:45:34.340126 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:45:34.340431 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:45:34.340438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:45:34.340443 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 02:45:34.340638 (+ 195us) client_negotiation.cc:240] Negotiation successful
0621 02:45:34.340872 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":258,"threads_started":1}
I20260621 02:45:34.343423 20015 heartbeater.cc:347] Connected to a master server at 127.9.166.62:40615
I20260621 02:45:34.343714 20015 heartbeater.cc:464] Registering TS with master...
I20260621 02:45:34.344359 20015 heartbeater.cc:511] Master 127.9.166.62:40615 requested a full tablet report, sending...
I20260621 02:45:34.345170 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:40615
--builtin_ntp_servers=127.9.166.20:38921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260621 02:45:34.345705 19698 ts_manager.cc:194] Registered new tserver with Master: 37f8f99c442c4fe781f6321b46f6958e (127.9.166.2:40831)
W20260621 02:45:34.677474 20018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:34.677780 20018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:34.677947 20018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:34.685096 20018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:34.685302 20018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:34.685478 20018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:45:34.694130 20018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:38921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:40615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:34.695487 20018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:34.697199 20018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:34.709522 20023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:34.709617 20024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:34.709601 20026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:34.710366 20018 server_base.cc:1034] running on GCE node
I20260621 02:45:34.710974 20018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:34.712217 20018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:34.713416 20018 hybrid_clock.cc:648] HybridClock initialized: now 1782009934713377 us; error 28 us; skew 500 ppm
I20260621 02:45:34.713740 20018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:45:34.716001 20018 webserver.cc:460] Webserver started at http://127.9.166.3:46481/ using document root <none> and password file <none>
I20260621 02:45:34.716656 20018 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:34.716773 20018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:34.717094 20018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:45:34.719864 20018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "4bf98a457c4240de8d450d8e74588ef1"
format_stamp: "Formatted at 2026-06-21 02:45:34 on dist-test-slave-hvgn"
server_key: "170f8fc67f925eb8be55f5f699a6615b"
server_key_iv: "5ace839b92d2945242a07e493ba69955"
server_key_version: "encryptionkey@0"
I20260621 02:45:34.720598 20018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "4bf98a457c4240de8d450d8e74588ef1"
format_stamp: "Formatted at 2026-06-21 02:45:34 on dist-test-slave-hvgn"
server_key: "170f8fc67f925eb8be55f5f699a6615b"
server_key_iv: "5ace839b92d2945242a07e493ba69955"
server_key_version: "encryptionkey@0"
I20260621 02:45:34.725572 20018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:45:34.729122 20032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:34.730414 20018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:45:34.730580 20018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "4bf98a457c4240de8d450d8e74588ef1"
format_stamp: "Formatted at 2026-06-21 02:45:34 on dist-test-slave-hvgn"
server_key: "170f8fc67f925eb8be55f5f699a6615b"
server_key_iv: "5ace839b92d2945242a07e493ba69955"
server_key_version: "encryptionkey@0"
I20260621 02:45:34.730794 20018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:34.736279 20018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:34.737030 20018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:34.737294 20018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:34.738212 20018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:45:34.739677 20018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:45:34.739795 20018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:34.739923 20018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:45:34.740012 20018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:34.772852 20018 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:37427
I20260621 02:45:34.772939 20144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:37427 every 8 connection(s)
I20260621 02:45:34.774135 20018 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:45:34.776134 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 20018
I20260621 02:45:34.776352 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:45:34.776875 9880 external_mini_cluster.cc:1428] Setting key 3d25a5ec55b87492947fdfdcb38c4b71
I20260621 02:45:34.782869 19737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:34.776819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:58399 (local address 127.9.166.62:40615)
0621 02:45:34.777039 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:34.777047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:34.778566 (+ 1519us) server_negotiation.cc:407] Connection header received
0621 02:45:34.780685 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:34.780690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:34.780753 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:34.780842 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:34.782139 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:34.782145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:34.782149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:45:34.782187 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:34.782223 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:45:34.782234 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:34.782359 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:34.782560 (+ 201us) server_negotiation.cc:299] Negotiation successful
0621 02:45:34.782622 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:45:34.783778 20147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:34.777349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:40615 (local address 127.9.166.3:58399)
0621 02:45:34.778434 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:34.778470 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:34.780438 (+ 1968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:34.781054 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:34.781063 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:34.781192 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:34.781868 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:45:34.781871 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:34.781906 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:45:34.781983 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:45:34.782343 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:45:34.782353 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:45:34.782360 (+ 7us) client_negotiation.cc:769] Sending connection context
0621 02:45:34.782625 (+ 265us) client_negotiation.cc:240] Negotiation successful
0621 02:45:34.782866 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":335,"threads_started":1}
I20260621 02:45:34.785013 20145 heartbeater.cc:347] Connected to a master server at 127.9.166.62:40615
I20260621 02:45:34.785284 20145 heartbeater.cc:464] Registering TS with master...
I20260621 02:45:34.785802 20145 heartbeater.cc:511] Master 127.9.166.62:40615 requested a full tablet report, sending...
I20260621 02:45:34.787002 19698 ts_manager.cc:194] Registered new tserver with Master: 4bf98a457c4240de8d450d8e74588ef1 (127.9.166.3:37427)
I20260621 02:45:34.792102 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260621 02:45:34.797897 19737 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:45:34.796436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39180 (local address 127.9.166.62:40615)
0621 02:45:34.796600 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:34.796606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:34.797115 (+ 509us) server_negotiation.cc:407] Connection header received
0621 02:45:34.797226 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:34.797229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:34.797274 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:34.797347 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:34.797675 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39180: BlockingRecv error: recv got EOF from 127.0.0.1:39180 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:45:34.798403 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 19758
I20260621 02:45:34.801748 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:34.799316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37650 (local address 127.9.166.1:38859)
0621 02:45:34.799816 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:34.799823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:34.799844 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:45:34.799894 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:34.799899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:34.800064 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:34.800219 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:34.801113 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:34.801119 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:34.801128 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:45:34.801140 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:34.801178 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:45:34.801191 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:34.801344 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:34.801415 (+ 71us) server_negotiation.cc:299] Negotiation successful
0621 02:45:34.801514 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":245,"threads_started":1}
I20260621 02:45:34.802860 19880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:34.884418 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 19758
I20260621 02:45:34.900779 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 19888
I20260621 02:45:34.904342 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:34.901664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59928 (local address 127.9.166.2:40831)
0621 02:45:34.902196 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:34.902204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:34.902233 (+ 29us) server_negotiation.cc:407] Connection header received
0621 02:45:34.902302 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:34.902308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:34.902527 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:34.902654 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:34.903536 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:34.903544 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:34.903557 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:45:34.903577 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:34.903640 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:45:34.903662 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:34.903853 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:34.903930 (+ 77us) server_negotiation.cc:299] Negotiation successful
0621 02:45:34.904048 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":263,"threads_started":1}
I20260621 02:45:34.905568 20010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:34.985538 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 19888
I20260621 02:45:35.001295 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 20018
I20260621 02:45:35.005273 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:35.002175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51710 (local address 127.9.166.3:37427)
0621 02:45:35.002829 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:35.002836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:35.002865 (+ 29us) server_negotiation.cc:407] Connection header received
0621 02:45:35.002927 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:35.002934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:35.003152 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:35.003356 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:35.004517 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:35.004523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:35.004535 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:45:35.004552 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:35.004591 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:45:35.004604 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:35.004796 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:35.004876 (+ 80us) server_negotiation.cc:299] Negotiation successful
0621 02:45:35.004977 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":345,"threads_started":1}
I20260621 02:45:35.006520 20140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:35.088187 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 20018
I20260621 02:45:35.105367 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 19668
I20260621 02:45:35.106590 19729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:35.216925 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 19668
2026-06-21T02:45:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2274 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:45:35 dist-test-slave-hvgn krb5kdc[20163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:45:35 dist-test-slave-hvgn krb5kdc[20163](info): set up 2 sockets
Jun 21 02:45:35 dist-test-slave-hvgn krb5kdc[20163](info): commencing operation
krb5kdc: starting...
W20260621 02:45:36.324400 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:45:36 dist-test-slave-hvgn krb5kdc[20163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:45:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:45:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:45:36.538385 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:38703
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:38703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:45:36.855027 20176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:36.855372 20176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:36.855538 20176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:36.863216 20176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:45:36.863313 20176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:36.863359 20176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:36.863481 20176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:45:36.863530 20176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:45:36.873183 20176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:38703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:38703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:36.874418 20176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:36.875710 20176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:36.886868 20181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:36.886902 20182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:36.886958 20184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:36.887753 20176 server_base.cc:1034] running on GCE node
I20260621 02:45:36.888650 20176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:36.890050 20176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:36.891300 20176 hybrid_clock.cc:648] HybridClock initialized: now 1782009936891266 us; error 55 us; skew 500 ppm
I20260621 02:45:36.891706 20176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:45:36 dist-test-slave-hvgn krb5kdc[20163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009936, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:45:36.894798 20176 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:45:36.896277 20176 webserver.cc:460] Webserver started at http://127.9.166.62:40635/ using document root <none> and password file <none>
I20260621 02:45:36.897019 20176 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:36.897161 20176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:36.897462 20176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:45:36.900372 20176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "040cee98e98b46529f47dc8eb4cb18d7"
format_stamp: "Formatted at 2026-06-21 02:45:36 on dist-test-slave-hvgn"
server_key: "8a9ab43fdc5ca49d8bc34095c230196d"
server_key_iv: "3d4bd61462008deab4a31e9337852493"
server_key_version: "encryptionkey@0"
I20260621 02:45:36.901114 20176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "040cee98e98b46529f47dc8eb4cb18d7"
format_stamp: "Formatted at 2026-06-21 02:45:36 on dist-test-slave-hvgn"
server_key: "8a9ab43fdc5ca49d8bc34095c230196d"
server_key_iv: "3d4bd61462008deab4a31e9337852493"
server_key_version: "encryptionkey@0"
I20260621 02:45:36.906023 20176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:45:36.909449 20191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:36.910725 20176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:45:36.910904 20176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "040cee98e98b46529f47dc8eb4cb18d7"
format_stamp: "Formatted at 2026-06-21 02:45:36 on dist-test-slave-hvgn"
server_key: "8a9ab43fdc5ca49d8bc34095c230196d"
server_key_iv: "3d4bd61462008deab4a31e9337852493"
server_key_version: "encryptionkey@0"
I20260621 02:45:36.911130 20176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:36.940397 20176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:36.944401 20176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:36.944682 20176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:36.962829 20176 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:38703
I20260621 02:45:36.962859 20242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:38703 every 8 connection(s)
I20260621 02:45:36.964207 20176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:45:36.967795 20243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:36.968119 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 20176
I20260621 02:45:36.968364 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:45:36.968976 9880 external_mini_cluster.cc:1428] Setting key a0b09e15f6768eb7a1e96abfe81a3347
I20260621 02:45:36.981690 20243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7: Bootstrap starting.
Jun 21 02:45:36 dist-test-slave-hvgn krb5kdc[20163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:45:36.984804 20243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:36.985827 20243 log.cc:824] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7: Log is configured to *not* fsync() on all Append() calls
I20260621 02:45:36.988643 20243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7: No bootstrap required, opened a new log
I20260621 02:45:36.992237 20246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:36.971024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35142 (local address 127.9.166.62:38703)
0621 02:45:36.971829 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:36.971844 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:36.971892 (+ 48us) server_negotiation.cc:407] Connection header received
0621 02:45:36.974251 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:36.974280 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:36.974803 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:36.975244 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:36.976608 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:36.978600 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:36.979919 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:36.980939 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:36.981106 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:36.985126 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:36.985135 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:36.985149 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:36.985189 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:36.988251 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:36.988971 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:36.988977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:36.988983 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:36.989161 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:36.989537 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:36.989546 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:36.989548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:36.990046 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:36.990387 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:36.990790 (+ 403us) server_negotiation.cc:299] Negotiation successful
0621 02:45:36.991190 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":350,"threads_started":1}
I20260621 02:45:36.996582 20243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040cee98e98b46529f47dc8eb4cb18d7" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38703 } }
I20260621 02:45:36.996961 20243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:36.997066 20243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040cee98e98b46529f47dc8eb4cb18d7, State: Initialized, Role: FOLLOWER
I20260621 02:45:36.997587 20243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040cee98e98b46529f47dc8eb4cb18d7" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38703 } }
I20260621 02:45:36.997819 20243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:45:36.997959 20243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:45:36.998129 20243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:37.000545 20243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040cee98e98b46529f47dc8eb4cb18d7" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38703 } }
I20260621 02:45:37.001006 20243 leader_election.cc:302] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 040cee98e98b46529f47dc8eb4cb18d7; no voters:
I20260621 02:45:37.001551 20243 leader_election.cc:288] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:45:37.001647 20248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:45:37.002341 20248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7 [term 1 LEADER]: Becoming Leader. State: Replica: 040cee98e98b46529f47dc8eb4cb18d7, State: Running, Role: LEADER
I20260621 02:45:37.002823 20248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040cee98e98b46529f47dc8eb4cb18d7" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38703 } }
I20260621 02:45:37.003765 20243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:45:37.005672 20249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "040cee98e98b46529f47dc8eb4cb18d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040cee98e98b46529f47dc8eb4cb18d7" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38703 } } }
I20260621 02:45:37.006124 20249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7 [sys.catalog]: This master's current role is: LEADER
I20260621 02:45:37.007563 20250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 040cee98e98b46529f47dc8eb4cb18d7. Latest consensus state: current_term: 1 leader_uuid: "040cee98e98b46529f47dc8eb4cb18d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "040cee98e98b46529f47dc8eb4cb18d7" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38703 } } }
I20260621 02:45:37.007863 20250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7 [sys.catalog]: This master's current role is: LEADER
I20260621 02:45:37.009882 20256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:45:37.015134 20256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:45:37.024856 20256 catalog_manager.cc:1347] Generated new cluster ID: 9ade2976b9fc46a89ec202cdcd290180
I20260621 02:45:37.024962 20256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:45:37.041580 20256 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:45:37.043507 20256 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:45:37.066128 20256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7: Generated new TSK 0
I20260621 02:45:37.067126 20256 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
I20260621 02:45:37.124326 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:38703
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:45:37.447541 20269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:37.447806 20269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:37.447961 20269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:37.455323 20269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:37.455464 20269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:37.455634 20269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:45:37.465163 20269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:38703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:37.466415 20269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:37.467814 20269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:37.480481 20274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:37.480540 20275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:37.480518 20277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:37.481510 20269 server_base.cc:1034] running on GCE node
I20260621 02:45:37.482080 20269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:37.483107 20269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:37.484334 20269 hybrid_clock.cc:648] HybridClock initialized: now 1782009937484281 us; error 41 us; skew 500 ppm
I20260621 02:45:37.484653 20269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:45:37 dist-test-slave-hvgn krb5kdc[20163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009937, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:45:37.490876 20269 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:45:37.492518 20269 webserver.cc:460] Webserver started at http://127.9.166.1:36811/ using document root <none> and password file <none>
I20260621 02:45:37.493201 20269 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:37.493345 20269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:37.493700 20269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:45:37.496666 20269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "2888533c578f4227a62de72a7afa824f"
format_stamp: "Formatted at 2026-06-21 02:45:37 on dist-test-slave-hvgn"
server_key: "090ad9a2784f3ada6188c76886f73cbb"
server_key_iv: "0f9b41d3446190e2930c9f58726605b8"
server_key_version: "encryptionkey@0"
I20260621 02:45:37.497423 20269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "2888533c578f4227a62de72a7afa824f"
format_stamp: "Formatted at 2026-06-21 02:45:37 on dist-test-slave-hvgn"
server_key: "090ad9a2784f3ada6188c76886f73cbb"
server_key_iv: "0f9b41d3446190e2930c9f58726605b8"
server_key_version: "encryptionkey@0"
I20260621 02:45:37.502275 20269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:45:37.505800 20284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:37.507103 20269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:45:37.507289 20269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "2888533c578f4227a62de72a7afa824f"
format_stamp: "Formatted at 2026-06-21 02:45:37 on dist-test-slave-hvgn"
server_key: "090ad9a2784f3ada6188c76886f73cbb"
server_key_iv: "0f9b41d3446190e2930c9f58726605b8"
server_key_version: "encryptionkey@0"
I20260621 02:45:37.507544 20269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:37.541818 20269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:37.546264 20269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:37.546561 20269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:37.547716 20269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:45:37.549036 20269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:45:37.549136 20269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:37.549289 20269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:45:37.549397 20269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:37.582396 20269 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:42515
I20260621 02:45:37.582484 20396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:42515 every 8 connection(s)
I20260621 02:45:37.583792 20269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:45:37.585242 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 20269
I20260621 02:45:37.585482 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:45:37.586278 9880 external_mini_cluster.cc:1428] Setting key 2320f388526510f04ba2ed42acdd1691
Jun 21 02:45:37 dist-test-slave-hvgn krb5kdc[20163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009937, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:45:37.605937 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:37.587195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:42031 (local address 127.9.166.62:38703)
0621 02:45:37.587693 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:37.587699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:37.588730 (+ 1031us) server_negotiation.cc:407] Connection header received
0621 02:45:37.590499 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:37.590506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:37.590588 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:37.590694 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:37.592144 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:37.593286 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:37.594969 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:37.595371 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:37.595556 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:37.598861 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:37.598870 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:37.598878 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:37.598921 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:37.602479 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:37.603174 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:37.603182 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:37.603188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:37.603358 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:37.603854 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:37.603862 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:37.603869 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:37.604269 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:37.604422 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:37.605192 (+ 770us) server_negotiation.cc:299] Negotiation successful
0621 02:45:37.605454 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":242,"threads_started":1}
I20260621 02:45:37.606602 20400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:37.587797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38703 (local address 127.9.166.1:42031)
0621 02:45:37.588569 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:37.588613 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:37.590253 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:37.590899 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:37.590907 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:37.591446 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:37.591983 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:37.591999 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:37.593553 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:37.593557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:37.594795 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:37.594806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:37.595643 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:37.595666 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:37.595906 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:37.596564 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:37.596590 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:37.598648 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:45:37.602616 (+ 3968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:37.602622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:37.602633 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:37.603060 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:37.603503 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:37.603506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:37.603509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:37.603719 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:37.604401 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:45:37.604407 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:45:37.604731 (+ 324us) client_negotiation.cc:769] Sending connection context
0621 02:45:37.605055 (+ 324us) client_negotiation.cc:240] Negotiation successful
0621 02:45:37.605472 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":339,"threads_started":1}
I20260621 02:45:37.608246 20397 heartbeater.cc:347] Connected to a master server at 127.9.166.62:38703
I20260621 02:45:37.608560 20397 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.166.2@KRBTEST.COM; defaulting to no policy
I20260621 02:45:37.609324 20397 heartbeater.cc:511] Master 127.9.166.62:38703 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.2@KRBTEST.COM" created.
I20260621 02:45:37.611411 20208 ts_manager.cc:194] Registered new tserver with Master: 2888533c578f4227a62de72a7afa824f (127.9.166.1:42515)
I20260621 02:45:37.612914 20208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:42031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
I20260621 02:45:37.637708 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:38703
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:45:37.968642 20403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:37.968910 20403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:37.969071 20403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:37.976143 20403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:37.976245 20403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:37.976462 20403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:45:37.985759 20403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:38703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:37.986999 20403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:37.988412 20403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:38.001332 20408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:38.001430 20409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:38.001441 20411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:38.002707 20403 server_base.cc:1034] running on GCE node
I20260621 02:45:38.003355 20403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:38.004413 20403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:38.005628 20403 hybrid_clock.cc:648] HybridClock initialized: now 1782009938005567 us; error 52 us; skew 500 ppm
I20260621 02:45:38.006068 20403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:45:38 dist-test-slave-hvgn krb5kdc[20163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009938, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:45:38.009550 20403 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:45:38.011191 20403 webserver.cc:460] Webserver started at http://127.9.166.2:42857/ using document root <none> and password file <none>
I20260621 02:45:38.011951 20403 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:38.012110 20403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:38.012444 20403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:45:38.015602 20403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "871abd7a3c3a4a8183a6201a059237a7"
format_stamp: "Formatted at 2026-06-21 02:45:38 on dist-test-slave-hvgn"
server_key: "27746a88e00658673b5ad9d28920cecc"
server_key_iv: "f5c73b70d2f232c7e0d09c28b908011b"
server_key_version: "encryptionkey@0"
I20260621 02:45:38.016445 20403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "871abd7a3c3a4a8183a6201a059237a7"
format_stamp: "Formatted at 2026-06-21 02:45:38 on dist-test-slave-hvgn"
server_key: "27746a88e00658673b5ad9d28920cecc"
server_key_iv: "f5c73b70d2f232c7e0d09c28b908011b"
server_key_version: "encryptionkey@0"
I20260621 02:45:38.021654 20403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260621 02:45:38.025146 20418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:38.026500 20403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:45:38.026667 20403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "871abd7a3c3a4a8183a6201a059237a7"
format_stamp: "Formatted at 2026-06-21 02:45:38 on dist-test-slave-hvgn"
server_key: "27746a88e00658673b5ad9d28920cecc"
server_key_iv: "f5c73b70d2f232c7e0d09c28b908011b"
server_key_version: "encryptionkey@0"
I20260621 02:45:38.026880 20403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:38.046802 20403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:38.051200 20403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:38.051512 20403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:38.052485 20403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:45:38.053789 20403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:45:38.053861 20403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:38.054029 20403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:45:38.054078 20403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:38.087495 20403 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:44635
I20260621 02:45:38.087605 20530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:44635 every 8 connection(s)
I20260621 02:45:38.088744 20403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:45:38.089545 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 20403
I20260621 02:45:38.089735 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:45:38.090281 9880 external_mini_cluster.cc:1428] Setting key 0d5e40a2ca2c724d1170f3f8a30ae4e6
Jun 21 02:45:38 dist-test-slave-hvgn krb5kdc[20163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009938, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:45:38.109614 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:38.091633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:47039 (local address 127.9.166.62:38703)
0621 02:45:38.091863 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:38.091869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:38.093199 (+ 1330us) server_negotiation.cc:407] Connection header received
0621 02:45:38.094995 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:38.095003 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:38.095092 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:38.095200 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:38.096752 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.097909 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.099681 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.100101 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.100197 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:38.103672 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:38.103678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:38.103681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:38.103713 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:38.106678 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:38.107368 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:38.107374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:38.107377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:38.107544 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:38.107959 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:38.107964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:38.107967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:38.108238 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:38.108387 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:38.109059 (+ 672us) server_negotiation.cc:299] Negotiation successful
0621 02:45:38.109274 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:45:38.110605 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:38.092220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38703 (local address 127.9.166.2:47039)
0621 02:45:38.093030 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:38.093076 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:38.094738 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:38.095467 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:38.095476 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:38.095996 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:38.096583 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:38.096600 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.098037 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.098041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:38.099531 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:38.099547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.100314 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.100319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:38.100562 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:38.101231 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:38.101257 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:38.103367 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:45:38.106808 (+ 3441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:38.106817 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:38.106833 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:38.107240 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:38.107650 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:38.107656 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:38.107659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:38.107838 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:38.108360 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:45:38.108368 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:45:38.108692 (+ 324us) client_negotiation.cc:769] Sending connection context
0621 02:45:38.108951 (+ 259us) client_negotiation.cc:240] Negotiation successful
0621 02:45:38.109474 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":328,"threads_started":1}
I20260621 02:45:38.112143 20531 heartbeater.cc:347] Connected to a master server at 127.9.166.62:38703
I20260621 02:45:38.112497 20531 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.166.3@KRBTEST.COM; defaulting to no policy
I20260621 02:45:38.113330 20531 heartbeater.cc:511] Master 127.9.166.62:38703 requested a full tablet report, sending...
I20260621 02:45:38.114974 20208 ts_manager.cc:194] Registered new tserver with Master: 871abd7a3c3a4a8183a6201a059237a7 (127.9.166.2:44635)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.3@KRBTEST.COM" created.
I20260621 02:45:38.116027 20208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:47039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
I20260621 02:45:38.140302 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:38703
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:45:38.456660 20536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:45:38.456944 20536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:45:38.457044 20536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:45:38.464053 20536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:45:38.464169 20536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:45:38.464282 20536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:45:38.473888 20536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:38703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:45:38.475348 20536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:45:38.476799 20536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:45:38.489135 20541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:38.489092 20542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:45:38.489352 20544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:45:38.490165 20536 server_base.cc:1034] running on GCE node
I20260621 02:45:38.490721 20536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:45:38.491678 20536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:45:38.492866 20536 hybrid_clock.cc:648] HybridClock initialized: now 1782009938492805 us; error 54 us; skew 500 ppm
I20260621 02:45:38.493167 20536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:45:38 dist-test-slave-hvgn krb5kdc[20163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009938, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:45:38.496232 20536 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:45:38.497627 20536 webserver.cc:460] Webserver started at http://127.9.166.3:45431/ using document root <none> and password file <none>
I20260621 02:45:38.498296 20536 fs_manager.cc:359] Metadata directory not provided
I20260621 02:45:38.498435 20536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:45:38.498721 20536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:45:38.501618 20536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "db870f2bcdf64640b00751f3d8924ed1"
format_stamp: "Formatted at 2026-06-21 02:45:38 on dist-test-slave-hvgn"
server_key: "f9a6af720660fb05997b39195ba8917d"
server_key_iv: "edc555c49d199c6c89fe7469e3b2e889"
server_key_version: "encryptionkey@0"
I20260621 02:45:38.502414 20536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "db870f2bcdf64640b00751f3d8924ed1"
format_stamp: "Formatted at 2026-06-21 02:45:38 on dist-test-slave-hvgn"
server_key: "f9a6af720660fb05997b39195ba8917d"
server_key_iv: "edc555c49d199c6c89fe7469e3b2e889"
server_key_version: "encryptionkey@0"
I20260621 02:45:38.507375 20536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260621 02:45:38.510746 20551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:38.512390 20536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 02:45:38.512563 20536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "db870f2bcdf64640b00751f3d8924ed1"
format_stamp: "Formatted at 2026-06-21 02:45:38 on dist-test-slave-hvgn"
server_key: "f9a6af720660fb05997b39195ba8917d"
server_key_iv: "edc555c49d199c6c89fe7469e3b2e889"
server_key_version: "encryptionkey@0"
I20260621 02:45:38.512759 20536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:45:38.530023 20536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:45:38.534438 20536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:45:38.534713 20536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:45:38.535650 20536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:45:38.536996 20536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:45:38.537204 20536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:38.537398 20536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:45:38.537477 20536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:45:38.570485 20536 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:41367
I20260621 02:45:38.570533 20663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:41367 every 8 connection(s)
I20260621 02:45:38.571798 20536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:45:38.580163 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 20536
I20260621 02:45:38.580379 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:45:38.580924 9880 external_mini_cluster.cc:1428] Setting key d38c85582c4ad12fb35113337182bb57
Jun 21 02:45:38 dist-test-slave-hvgn krb5kdc[20163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009938, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:45:38.592545 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:38.574335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:37765 (local address 127.9.166.62:38703)
0621 02:45:38.574515 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:38.574520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:38.575696 (+ 1176us) server_negotiation.cc:407] Connection header received
0621 02:45:38.577453 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:38.577457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:38.577538 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:38.577628 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:38.579118 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.579977 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.582226 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.582834 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.582944 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:38.587056 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:38.587063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:38.587068 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:38.587108 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:38.589905 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:38.590624 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:38.590627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:38.590629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:38.590751 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:38.591104 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:38.591107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:38.591109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:38.591355 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:38.591534 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:38.592161 (+ 627us) server_negotiation.cc:299] Negotiation successful
0621 02:45:38.592313 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:45:38.593473 20666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:38.574787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38703 (local address 127.9.166.3:37765)
0621 02:45:38.575535 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:38.575579 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:38.577184 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:38.577844 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:38.577852 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:38.578364 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:45:38.578911 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:38.578927 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.580131 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.580138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:38.582053 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:38.582068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.583330 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.583338 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:38.583654 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:38.584328 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:45:38.584362 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:45:38.586839 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:45:38.590028 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:38.590034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:38.590045 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:38.590527 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:38.590844 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:45:38.590847 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:45:38.590850 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:45:38.591017 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:45:38.591522 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:45:38.591528 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:45:38.591799 (+ 271us) client_negotiation.cc:769] Sending connection context
0621 02:45:38.592059 (+ 260us) client_negotiation.cc:240] Negotiation successful
0621 02:45:38.592405 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":260,"threads_started":1}
I20260621 02:45:38.595135 20664 heartbeater.cc:347] Connected to a master server at 127.9.166.62:38703
I20260621 02:45:38.595475 20664 heartbeater.cc:464] Registering TS with master...
I20260621 02:45:38.596436 20664 heartbeater.cc:511] Master 127.9.166.62:38703 requested a full tablet report, sending...
I20260621 02:45:38.597761 20208 ts_manager.cc:194] Registered new tserver with Master: db870f2bcdf64640b00751f3d8924ed1 (127.9.166.3:41367)
I20260621 02:45:38.598556 20208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:37765
I20260621 02:45:38.608484 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:45:38.615891 20397 heartbeater.cc:503] Master 127.9.166.62:38703 was elected leader, sending a full tablet report...
I20260621 02:45:38.624993 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:38.612854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35150 (local address 127.9.166.62:38703)
0621 02:45:38.613016 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:38.613021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:38.613486 (+ 465us) server_negotiation.cc:407] Connection header received
0621 02:45:38.613607 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:38.613610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:38.613668 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:38.613751 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:38.615276 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.616321 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.617788 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.618167 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.618258 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:38.620342 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:38.620346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:38.620349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:38.620377 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:38.622839 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:38.623459 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:38.623463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:38.623465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:38.623579 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:38.623879 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:38.623887 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:38.623889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:38.624190 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:38.624366 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:38.624613 (+ 247us) server_negotiation.cc:299] Negotiation successful
0621 02:45:38.624754 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:45:38.628597 20208 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:45:38.635298 20208 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:45:38.659412 20676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:38.646723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:42515 (local address 127.0.0.1:52432)
0621 02:45:38.647595 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:38.647652 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:38.647860 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:38.649459 (+ 1599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:38.649470 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:38.649507 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:45:38.650034 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:38.650052 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.652839 (+ 2787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.652848 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:38.655992 (+ 3144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:38.656019 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.658406 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.658413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:38.658615 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:38.658677 (+ 62us) client_negotiation.cc:769] Sending connection context
0621 02:45:38.658790 (+ 113us) client_negotiation.cc:240] Negotiation successful
0621 02:45:38.659008 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":469,"threads_started":1}
I20260621 02:45:38.660573 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:38.647202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52432 (local address 127.9.166.1:42515)
0621 02:45:38.648685 (+ 1483us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:38.648697 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:38.648732 (+ 35us) server_negotiation.cc:407] Connection header received
0621 02:45:38.648845 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:38.648856 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:38.649179 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:38.650273 (+ 1094us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:45:38.650566 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.652650 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.656271 (+ 3621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.658204 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.659764 (+ 1560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:38.660022 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:38.660087 (+ 65us) server_negotiation.cc:299] Negotiation successful
0621 02:45:38.660243 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1272,"thread_start_us":1102,"threads_started":1}
I20260621 02:45:38.664361 20332 tablet_service.cc:1467] Processing CreateTablet for tablet 025196c6cd234a279d50a0dbb7f9ffd3 (DEFAULT_TABLE table=test-table [id=b89c5ec97ee6474a80b932317a3ceb83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:45:38.665962 20332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 025196c6cd234a279d50a0dbb7f9ffd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:38.666759 20678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:38.647271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:41367 (local address 127.0.0.1:53908)
0621 02:45:38.658050 (+ 10779us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:38.658078 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:38.658213 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:38.660331 (+ 2118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:38.660335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:38.660370 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:45:38.660701 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:38.660715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.662754 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.662759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:38.664367 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:38.664381 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.665925 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.665931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:38.666100 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:38.666137 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 02:45:38.666264 (+ 127us) client_negotiation.cc:240] Negotiation successful
0621 02:45:38.666386 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10612,"mutex_wait_us":1,"thread_start_us":10436,"threads_started":1}
I20260621 02:45:38.666779 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:38.647786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.9.166.3:41367)
0621 02:45:38.659689 (+ 11903us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:38.659698 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:38.659730 (+ 32us) server_negotiation.cc:407] Connection header received
0621 02:45:38.659817 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:38.659826 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:38.660120 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:38.660350 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:45:38.660928 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.662593 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.664549 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.665708 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.665861 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:38.666078 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:38.666311 (+ 233us) server_negotiation.cc:299] Negotiation successful
0621 02:45:38.666464 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11729,"thread_start_us":12230,"threads_started":1}
I20260621 02:45:38.667907 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:38.648083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:44635 (local address 127.0.0.1:46776)
0621 02:45:38.659660 (+ 11577us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:38.659685 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:38.659790 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:38.660639 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:38.660645 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:38.660676 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:45:38.661092 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:38.661104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.662948 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.662953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:38.665149 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:38.665162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.667171 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.667177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:38.667335 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:38.667366 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 02:45:38.667489 (+ 123us) client_negotiation.cc:240] Negotiation successful
0621 02:45:38.667602 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11447,"thread_start_us":11253,"threads_started":1}
I20260621 02:45:38.668037 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:38.648581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46776 (local address 127.9.166.2:44635)
0621 02:45:38.659184 (+ 10603us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:38.659193 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:38.659900 (+ 707us) server_negotiation.cc:407] Connection header received
0621 02:45:38.660007 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:38.660014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:38.660245 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:38.660446 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:45:38.661302 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.662774 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.665332 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.666742 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.666858 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:38.667028 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:38.667519 (+ 491us) server_negotiation.cc:299] Negotiation successful
0621 02:45:38.667684 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10424,"thread_start_us":10306,"threads_started":1}
I20260621 02:45:38.670257 20599 tablet_service.cc:1467] Processing CreateTablet for tablet 025196c6cd234a279d50a0dbb7f9ffd3 (DEFAULT_TABLE table=test-table [id=b89c5ec97ee6474a80b932317a3ceb83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:45:38.671473 20599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 025196c6cd234a279d50a0dbb7f9ffd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:38.671670 20466 tablet_service.cc:1467] Processing CreateTablet for tablet 025196c6cd234a279d50a0dbb7f9ffd3 (DEFAULT_TABLE table=test-table [id=b89c5ec97ee6474a80b932317a3ceb83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:45:38.672916 20466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 025196c6cd234a279d50a0dbb7f9ffd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:45:38.680934 20682 tablet_bootstrap.cc:492] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f: Bootstrap starting.
I20260621 02:45:38.684144 20683 tablet_bootstrap.cc:492] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1: Bootstrap starting.
I20260621 02:45:38.684613 20682 tablet_bootstrap.cc:654] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:38.685618 20682 log.cc:824] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f: Log is configured to *not* fsync() on all Append() calls
I20260621 02:45:38.687831 20683 tablet_bootstrap.cc:654] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:38.688766 20683 log.cc:824] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1: Log is configured to *not* fsync() on all Append() calls
I20260621 02:45:38.688918 20685 tablet_bootstrap.cc:492] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7: Bootstrap starting.
I20260621 02:45:38.688987 20682 tablet_bootstrap.cc:492] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f: No bootstrap required, opened a new log
I20260621 02:45:38.689227 20682 ts_tablet_manager.cc:1397] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260621 02:45:38.691108 20683 tablet_bootstrap.cc:492] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1: No bootstrap required, opened a new log
I20260621 02:45:38.691363 20683 ts_tablet_manager.cc:1397] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260621 02:45:38.692277 20685 tablet_bootstrap.cc:654] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7: Neither blocks nor log segments found. Creating new log.
I20260621 02:45:38.693434 20685 log.cc:824] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7: Log is configured to *not* fsync() on all Append() calls
I20260621 02:45:38.696689 20685 tablet_bootstrap.cc:492] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7: No bootstrap required, opened a new log
I20260621 02:45:38.696931 20685 ts_tablet_manager.cc:1397] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260621 02:45:38.698975 20683 raft_consensus.cc:348] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2888533c578f4227a62de72a7afa824f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42515 } } peers { permanent_uuid: "871abd7a3c3a4a8183a6201a059237a7" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44635 } } peers { permanent_uuid: "db870f2bcdf64640b00751f3d8924ed1" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41367 } }
I20260621 02:45:38.699594 20683 raft_consensus.cc:374] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:38.699715 20683 raft_consensus.cc:729] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db870f2bcdf64640b00751f3d8924ed1, State: Initialized, Role: FOLLOWER
I20260621 02:45:38.699474 20682 raft_consensus.cc:348] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2888533c578f4227a62de72a7afa824f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42515 } } peers { permanent_uuid: "871abd7a3c3a4a8183a6201a059237a7" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44635 } } peers { permanent_uuid: "db870f2bcdf64640b00751f3d8924ed1" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41367 } }
I20260621 02:45:38.699872 20682 raft_consensus.cc:374] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:38.699971 20682 raft_consensus.cc:729] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2888533c578f4227a62de72a7afa824f, State: Initialized, Role: FOLLOWER
I20260621 02:45:38.700454 20683 consensus_queue.cc:260] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2888533c578f4227a62de72a7afa824f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42515 } } peers { permanent_uuid: "871abd7a3c3a4a8183a6201a059237a7" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44635 } } peers { permanent_uuid: "db870f2bcdf64640b00751f3d8924ed1" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41367 } }
I20260621 02:45:38.700546 20682 consensus_queue.cc:260] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2888533c578f4227a62de72a7afa824f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42515 } } peers { permanent_uuid: "871abd7a3c3a4a8183a6201a059237a7" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44635 } } peers { permanent_uuid: "db870f2bcdf64640b00751f3d8924ed1" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41367 } }
I20260621 02:45:38.702242 20682 ts_tablet_manager.cc:1428] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260621 02:45:38.703437 20664 heartbeater.cc:503] Master 127.9.166.62:38703 was elected leader, sending a full tablet report...
I20260621 02:45:38.704926 20683 ts_tablet_manager.cc:1428] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1: Time spent starting tablet: real 0.013s user 0.009s sys 0.002s
I20260621 02:45:38.706823 20685 raft_consensus.cc:348] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2888533c578f4227a62de72a7afa824f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42515 } } peers { permanent_uuid: "871abd7a3c3a4a8183a6201a059237a7" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44635 } } peers { permanent_uuid: "db870f2bcdf64640b00751f3d8924ed1" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41367 } }
I20260621 02:45:38.707228 20685 raft_consensus.cc:374] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:45:38.707432 20685 raft_consensus.cc:729] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 871abd7a3c3a4a8183a6201a059237a7, State: Initialized, Role: FOLLOWER
I20260621 02:45:38.708104 20685 consensus_queue.cc:260] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2888533c578f4227a62de72a7afa824f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42515 } } peers { permanent_uuid: "871abd7a3c3a4a8183a6201a059237a7" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44635 } } peers { permanent_uuid: "db870f2bcdf64640b00751f3d8924ed1" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41367 } }
I20260621 02:45:38.709190 20531 heartbeater.cc:503] Master 127.9.166.62:38703 was elected leader, sending a full tablet report...
I20260621 02:45:38.709753 20685 ts_tablet_manager.cc:1428] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260621 02:45:38.780917 20688 raft_consensus.cc:482] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:45:38.781189 20688 raft_consensus.cc:504] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2888533c578f4227a62de72a7afa824f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42515 } } peers { permanent_uuid: "871abd7a3c3a4a8183a6201a059237a7" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44635 } } peers { permanent_uuid: "db870f2bcdf64640b00751f3d8924ed1" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41367 } }
I20260621 02:45:38.782748 20688 leader_election.cc:288] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2888533c578f4227a62de72a7afa824f (127.9.166.1:42515), 871abd7a3c3a4a8183a6201a059237a7 (127.9.166.2:44635)
I20260621 02:45:38.789283 20666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:38.782989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:42515 (local address 127.9.166.3:56369)
0621 02:45:38.783213 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:38.783235 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:38.783362 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:38.783893 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:38.783901 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:38.783934 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:45:38.784148 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:38.784160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.785407 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.785410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:38.787341 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:38.787356 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.788688 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.788692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:38.788825 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:38.788849 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 02:45:38.788939 (+ 90us) client_negotiation.cc:240] Negotiation successful
0621 02:45:38.789028 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":1}
I20260621 02:45:38.789373 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:38.783104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:56369 (local address 127.9.166.1:42515)
0621 02:45:38.783326 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:38.783332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:38.783352 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:45:38.783541 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:38.783549 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:38.783652 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:38.783764 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:45:38.784289 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.785279 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.787518 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.788532 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.788662 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:38.788764 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:38.788986 (+ 222us) server_negotiation.cc:299] Negotiation successful
0621 02:45:38.789118 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 02:45:38.790102 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:38.783555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:58757 (local address 127.9.166.2:44635)
0621 02:45:38.783793 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:38.783800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:38.784141 (+ 341us) server_negotiation.cc:407] Connection header received
0621 02:45:38.784297 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:38.784302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:38.784381 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:38.784473 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:45:38.784930 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.785747 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.787978 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.789139 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.789266 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:38.789351 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:38.789749 (+ 398us) server_negotiation.cc:299] Negotiation successful
0621 02:45:38.789871 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260621 02:45:38.790141 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:38.783364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:44635 (local address 127.9.166.3:58757)
0621 02:45:38.784028 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0621 02:45:38.784045 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:45:38.784160 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:45:38.784495 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:45:38.784498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:45:38.784517 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:45:38.784776 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:38.784783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.785929 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.785934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:38.787828 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:45:38.787838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.789353 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.789360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:45:38.789566 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:38.789605 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 02:45:38.789699 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 02:45:38.789793 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":26,"thread_start_us":344,"threads_started":1}
I20260621 02:45:38.790522 20352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "025196c6cd234a279d50a0dbb7f9ffd3" candidate_uuid: "db870f2bcdf64640b00751f3d8924ed1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2888533c578f4227a62de72a7afa824f" is_pre_election: true
I20260621 02:45:38.791128 20352 raft_consensus.cc:2454] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db870f2bcdf64640b00751f3d8924ed1 in term 0.
I20260621 02:45:38.791139 20486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "025196c6cd234a279d50a0dbb7f9ffd3" candidate_uuid: "db870f2bcdf64640b00751f3d8924ed1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "871abd7a3c3a4a8183a6201a059237a7" is_pre_election: true
I20260621 02:45:38.791658 20486 raft_consensus.cc:2454] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db870f2bcdf64640b00751f3d8924ed1 in term 0.
I20260621 02:45:38.791899 20552 leader_election.cc:302] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2888533c578f4227a62de72a7afa824f, db870f2bcdf64640b00751f3d8924ed1; no voters:
I20260621 02:45:38.792250 20688 raft_consensus.cc:2781] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:45:38.792382 20688 raft_consensus.cc:482] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:45:38.792502 20688 raft_consensus.cc:3037] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:38.794617 20688 raft_consensus.cc:504] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2888533c578f4227a62de72a7afa824f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42515 } } peers { permanent_uuid: "871abd7a3c3a4a8183a6201a059237a7" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44635 } } peers { permanent_uuid: "db870f2bcdf64640b00751f3d8924ed1" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41367 } }
I20260621 02:45:38.795145 20688 leader_election.cc:288] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [CANDIDATE]: Term 1 election: Requested vote from peers 2888533c578f4227a62de72a7afa824f (127.9.166.1:42515), 871abd7a3c3a4a8183a6201a059237a7 (127.9.166.2:44635)
I20260621 02:45:38.795584 20352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "025196c6cd234a279d50a0dbb7f9ffd3" candidate_uuid: "db870f2bcdf64640b00751f3d8924ed1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2888533c578f4227a62de72a7afa824f"
I20260621 02:45:38.795693 20486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "025196c6cd234a279d50a0dbb7f9ffd3" candidate_uuid: "db870f2bcdf64640b00751f3d8924ed1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "871abd7a3c3a4a8183a6201a059237a7"
I20260621 02:45:38.795778 20352 raft_consensus.cc:3037] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:38.795915 20486 raft_consensus.cc:3037] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:45:38.797941 20352 raft_consensus.cc:2454] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db870f2bcdf64640b00751f3d8924ed1 in term 1.
I20260621 02:45:38.798305 20552 leader_election.cc:302] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2888533c578f4227a62de72a7afa824f, db870f2bcdf64640b00751f3d8924ed1; no voters:
I20260621 02:45:38.798384 20486 raft_consensus.cc:2454] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db870f2bcdf64640b00751f3d8924ed1 in term 1.
I20260621 02:45:38.798586 20688 raft_consensus.cc:2781] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:45:38.799064 20688 raft_consensus.cc:686] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [term 1 LEADER]: Becoming Leader. State: Replica: db870f2bcdf64640b00751f3d8924ed1, State: Running, Role: LEADER
I20260621 02:45:38.799612 20688 consensus_queue.cc:237] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2888533c578f4227a62de72a7afa824f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42515 } } peers { permanent_uuid: "871abd7a3c3a4a8183a6201a059237a7" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44635 } } peers { permanent_uuid: "db870f2bcdf64640b00751f3d8924ed1" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41367 } }
I20260621 02:45:38.804077 20207 catalog_manager.cc:5495] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 reported cstate change: term changed from 0 to 1, leader changed from <none> to db870f2bcdf64640b00751f3d8924ed1 (127.9.166.3). New cstate: current_term: 1 leader_uuid: "db870f2bcdf64640b00751f3d8924ed1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2888533c578f4227a62de72a7afa824f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "871abd7a3c3a4a8183a6201a059237a7" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db870f2bcdf64640b00751f3d8924ed1" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41367 } health_report { overall_health: HEALTHY } } }
I20260621 02:45:38.821622 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:38.816446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53918 (local address 127.9.166.3:41367)
0621 02:45:38.816693 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:38.816699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:38.816870 (+ 171us) server_negotiation.cc:407] Connection header received
0621 02:45:38.817044 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:38.817048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:38.817127 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:38.817260 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:45:38.817647 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.818384 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.819780 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.820164 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.820257 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:38.820688 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:45:38.821164 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:45:38.821270 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:38.821358 (+ 88us) server_negotiation.cc:299] Negotiation successful
0621 02:45:38.821425 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260621 02:45:38.823428 20665 tablet.cc:2369] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:45:38.829788 20486 raft_consensus.cc:1261] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7 [term 1 FOLLOWER]: Refusing update from remote peer db870f2bcdf64640b00751f3d8924ed1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:45:38.829825 20352 raft_consensus.cc:1261] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f [term 1 FOLLOWER]: Refusing update from remote peer db870f2bcdf64640b00751f3d8924ed1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:45:38.830824 20692 consensus_queue.cc:1035] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "871abd7a3c3a4a8183a6201a059237a7" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:45:38.831292 20688 consensus_queue.cc:1035] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2888533c578f4227a62de72a7afa824f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 42515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260621 02:45:38.836061 20398 tablet.cc:2369] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:45:38.841418 20532 tablet.cc:2369] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:45:38.855985 20208 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:45:38.856312 20208 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:45:38.860291 20208 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 040cee98e98b46529f47dc8eb4cb18d7: Sending DeleteTablet for 3 replicas of tablet 025196c6cd234a279d50a0dbb7f9ffd3
I20260621 02:45:38.861464 20599 tablet_service.cc:1514] Processing DeleteTablet for tablet 025196c6cd234a279d50a0dbb7f9ffd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:38 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:53908
I20260621 02:45:38.861480 20332 tablet_service.cc:1514] Processing DeleteTablet for tablet 025196c6cd234a279d50a0dbb7f9ffd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:38 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:52432
I20260621 02:45:38.862495 20466 tablet_service.cc:1514] Processing DeleteTablet for tablet 025196c6cd234a279d50a0dbb7f9ffd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:45:38 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:46776
I20260621 02:45:38.865104 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 20269
I20260621 02:45:38.874789 20707 tablet_replica.cc:330] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7: stopping tablet replica
I20260621 02:45:38.875370 20707 raft_consensus.cc:2229] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:45:38.876032 20707 raft_consensus.cc:2258] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:38.878244 20707 ts_tablet_manager.cc:1905] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:45:38.886832 20707 ts_tablet_manager.cc:1918] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:45:38.887086 20707 log.cc:1196] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/025196c6cd234a279d50a0dbb7f9ffd3
I20260621 02:45:38.887845 20707 ts_tablet_manager.cc:1939] T 025196c6cd234a279d50a0dbb7f9ffd3 P 871abd7a3c3a4a8183a6201a059237a7: Deleting consensus metadata
I20260621 02:45:38.889829 20192 catalog_manager.cc:4841] TS 871abd7a3c3a4a8183a6201a059237a7 (127.9.166.2:44635): tablet 025196c6cd234a279d50a0dbb7f9ffd3 (table test-table [id=b89c5ec97ee6474a80b932317a3ceb83]) successfully deleted
I20260621 02:45:38.891556 20705 tablet_replica.cc:330] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1: stopping tablet replica
I20260621 02:45:38.892153 20705 raft_consensus.cc:2229] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:45:38.892975 20705 raft_consensus.cc:2258] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:38.894413 20706 tablet_replica.cc:330] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f: stopping tablet replica
I20260621 02:45:38.894804 20706 raft_consensus.cc:2229] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:45:38.895262 20706 raft_consensus.cc:2258] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:45:38.895349 20705 ts_tablet_manager.cc:1905] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:45:38.897418 20706 ts_tablet_manager.cc:1905] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 21 02:45:38 dist-test-slave-hvgn krb5kdc[20163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:45:38.903990 20705 ts_tablet_manager.cc:1918] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:45:38.904202 20705 log.cc:1196] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/025196c6cd234a279d50a0dbb7f9ffd3
I20260621 02:45:38.904603 20706 ts_tablet_manager.cc:1918] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:45:38.904799 20706 log.cc:1196] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/025196c6cd234a279d50a0dbb7f9ffd3
I20260621 02:45:38.904896 20705 ts_tablet_manager.cc:1939] T 025196c6cd234a279d50a0dbb7f9ffd3 P db870f2bcdf64640b00751f3d8924ed1: Deleting consensus metadata
I20260621 02:45:38.905341 20706 ts_tablet_manager.cc:1939] T 025196c6cd234a279d50a0dbb7f9ffd3 P 2888533c578f4227a62de72a7afa824f: Deleting consensus metadata
I20260621 02:45:38.906766 20193 catalog_manager.cc:4841] TS db870f2bcdf64640b00751f3d8924ed1 (127.9.166.3:41367): tablet 025196c6cd234a279d50a0dbb7f9ffd3 (table test-table [id=b89c5ec97ee6474a80b932317a3ceb83]) successfully deleted
I20260621 02:45:38.907089 20192 catalog_manager.cc:4841] TS 2888533c578f4227a62de72a7afa824f (127.9.166.1:42515): tablet 025196c6cd234a279d50a0dbb7f9ffd3 (table test-table [id=b89c5ec97ee6474a80b932317a3ceb83]) successfully deleted
I20260621 02:45:38.907716 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:38.866079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52442 (local address 127.9.166.1:42515)
0621 02:45:38.866281 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:38.866287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:38.888771 (+ 22484us) server_negotiation.cc:407] Connection header received
0621 02:45:38.889285 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:38.889291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:38.889384 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:38.889471 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:38.891034 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.892237 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.894020 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:38.894698 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:38.894807 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:38.899952 (+ 5145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:38.899966 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:38.899983 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:38.900029 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:38.904192 (+ 4163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:38.905227 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:38.905235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:38.905238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:38.905403 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:38.905856 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:38.905861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:38.905864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:38.906302 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:38.906616 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:38.907180 (+ 564us) server_negotiation.cc:299] Negotiation successful
0621 02:45:38.907365 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:45:38.908953 20392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:39.010874 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 20269
I20260621 02:45:39.034329 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 20403
Jun 21 02:45:39 dist-test-slave-hvgn krb5kdc[20163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.2@KRBTEST.COM
I20260621 02:45:39.051602 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:39.035250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46784 (local address 127.9.166.2:44635)
0621 02:45:39.035500 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:39.035504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:39.035524 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:45:39.035700 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:39.035705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:39.035776 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:39.035857 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:39.037710 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:39.038546 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:39.040391 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:39.040788 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:39.040887 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:39.045792 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:39.045805 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:39.045820 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:39.045854 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:39.048857 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:39.049530 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:39.049537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:39.049539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:39.049661 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:39.050045 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:39.050054 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:39.050056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:39.050428 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:39.050717 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:39.051016 (+ 299us) server_negotiation.cc:299] Negotiation successful
0621 02:45:39.051236 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:45:39.052335 20526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:39.140333 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 20403
I20260621 02:45:39.163137 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 20536
Jun 21 02:45:39 dist-test-slave-hvgn krb5kdc[20163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.3@KRBTEST.COM
I20260621 02:45:39.181130 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:45:39.164102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53932 (local address 127.9.166.3:41367)
0621 02:45:39.164320 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 02:45:39.164326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:45:39.164346 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:45:39.164417 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:45:39.164423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:45:39.164537 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:45:39.164656 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:45:39.165867 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:39.167075 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:39.168666 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:45:39.169369 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:45:39.169505 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:45:39.173749 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:45:39.173763 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:45:39.173783 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:45:39.173836 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:45:39.178179 (+ 4343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:39.178859 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:39.178869 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:39.178873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:39.179076 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:45:39.179511 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:45:39.179517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:45:39.179521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:45:39.179983 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:45:39.180299 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0621 02:45:39.180489 (+ 190us) server_negotiation.cc:299] Negotiation successful
0621 02:45:39.180730 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 02:45:39.181892 20659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:39.307544 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 20536
I20260621 02:45:39.327610 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 20176
I20260621 02:45:39.328709 20238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:45:39.436967 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 20176
2026-06-21T02:45:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4221 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:45:39 dist-test-slave-hvgn krb5kdc[20715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:45:39 dist-test-slave-hvgn krb5kdc[20715](info): set up 2 sockets
Jun 21 02:45:39 dist-test-slave-hvgn krb5kdc[20715](info): commencing operation
krb5kdc: starting...
W20260621 02:45:40.580266 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:45:40 dist-test-slave-hvgn krb5kdc[20715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:45:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:45:40Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.166.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.166.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.166.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/rangeradmin_127.9.166.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.166.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/rangeradmin_127.9.166.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.166.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.166.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.166.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/rangerlookup_127.9.166.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.166.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/rangerlookup_127.9.166.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.166.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.166.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.166.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/HTTP_127.9.166.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.166.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/HTTP_127.9.166.20@KRBTEST.COM.keytab.
I20260621 02:45:40.882953 9880 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task2e3lzq/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/postgres -l logfile start
2026-06-21 02:45:43.205 UTC [20745] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-21 02:45:43.205 UTC [20745] LOG: listening on IPv4 address "127.9.166.20", port 49667
2026-06-21 02:45:43.208 UTC [20745] LOG: listening on Unix socket "/tmp/.s.PGSQL.49667"
2026-06-21 02:45:43.228 UTC [20748] LOG: database system was shut down at 2026-06-21 02:45:41 UTC
2026-06-21 02:45:43.231 UTC [20745] LOG: database system is ready to accept connections
I20260621 02:45:44.200286 9880 mini_postgres.cc:96] Postgres bound to 49667
2026-06-21 02:45:44.217 UTC [20756] FATAL: database "slave" does not exist
127.9.166.20:49667 - accepting connections
I20260621 02:45:44.218087 9880 mini_ranger.cc:161] Starting Ranger...
I20260621 02:45:44.242761 9880 mini_ranger.cc:84] Created miniranger Postgres user
I20260621 02:45:44.660972 9880 mini_ranger.cc:87] Created ranger Postgres database
I20260621 02:45:44.661087 9880 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ranger-admin
2026-06-21 02:45:44,945 [I] DB FLAVOR :POSTGRES
2026-06-21 02:45:44,946 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 02:45:44,946 [I] Checking connection..
2026-06-21 02:45:44,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-21 02:45:45,202 [I] Checking connection passed.
2026-06-21 02:45:45,202 [I] --------- Verifying version history table ---------
2026-06-21 02:45:45,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 02:45:45,433 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 02:45:45,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 02:45:45,661 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 02:45:45,661 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-21 02:45:45,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-21 02:45:45.883 UTC [20820] WARNING: there is no transaction in progress
2026-06-21 02:45:45,890 [I] create_dbversion_catalog.sql file imported successfully
2026-06-21 02:45:45,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 02:45:46,134 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-21 02:45:46,135 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 02:45:46,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-21 02:45:46,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-21 02:45:46,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hvgn.c.gcp-upstream.internal','N') ;"
2026-06-21 02:45:46,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 02:45:47,084 [I] Table x_portal_user does not exist in database ranger
2026-06-21 02:45:47,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 02:45:47,336 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-21 02:45:47,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 02:45:47,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 02:45:47,797 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-21 02:45:47,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-21 02:45:48.695 UTC [20928] WARNING: there is no transaction in progress
2026-06-21 02:45:48.711 UTC [20928] WARNING: there is no transaction in progress
2026-06-21 02:45:48.726 UTC [20928] WARNING: there is no transaction in progress
2026-06-21 02:45:48.740 UTC [20928] WARNING: there is no transaction in progress
2026-06-21 02:45:48.754 UTC [20928] WARNING: there is no transaction in progress
2026-06-21 02:45:48.770 UTC [20928] WARNING: there is no transaction in progress
2026-06-21 02:45:48.822 UTC [20928] WARNING: there is no transaction in progress
2026-06-21 02:45:48.828 UTC [20928] WARNING: there is no transaction in progress
2026-06-21 02:45:48.841 UTC [20928] WARNING: there is no transaction in progress
2026-06-21 02:45:48.851 UTC [20928] WARNING: there is no transaction in progress
2026-06-21 02:45:48.861 UTC [20928] WARNING: there is no transaction in progress
2026-06-21 02:45:48.874 UTC [20928] WARNING: there is no transaction in progress
2026-06-21 02:45:48.884 UTC [20928] WARNING: there is no transaction in progress
2026-06-21 02:45:48.890 UTC [20928] WARNING: there is no transaction in progress
2026-06-21 02:45:48.900 UTC [20928] WARNING: there is no transaction in progress
2026-06-21 02:45:48.908 UTC [20928] WARNING: there is no transaction in progress
2026-06-21 02:45:49,480 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-21 02:45:49,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-21 02:45:49,712 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-21 02:45:49,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 02:45:49,953 [I] Table x_portal_user already exists in database 'ranger'
2026-06-21 02:45:49,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 02:45:50,210 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-21 02:45:50,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 02:45:50,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 02:45:50,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hvgn.c.gcp-upstream.internal';"
2026-06-21 02:45:50,908 [I] CORE_DB_SCHEMA import status has been updated
2026-06-21 02:45:50,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-21 02:45:51,137 [I] DB_PATCHES have already been applied
I20260621 02:45:51.143801 9880 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ranger-admin:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task2e3lzq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260621 02:45:51.143886 9880 mini_ranger.cc:193] Using host: 127.9.166.20
I20260621 02:45:51.158370 9880 mini_ranger.cc:227] Ranger admin URL: http://127.9.166.20:58783
Jun 21, 2026 2:45:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 2:45:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 2:45:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ranger-admin/ews/webapp
Jun 21, 2026 2:45:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 21, 2026 2:45:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 21, 2026 2:45:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ranger-admin/ews/webapp].
Jun 21, 2026 2:45:52 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260621 02:45:52.260993 9880 mini_ranger.cc:160] Time spent starting Ranger: real 8.043s user 0.000s sys 0.037s
Jun 21, 2026 2:46:11 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 21, 2026 2:46:11 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 21, 2026 2:46:11 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 21, 2026 2:46:11 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 21, 2026 2:46:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 21, 2026 2:46:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 21, 2026 2:46:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 21, 2026 2:46:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 21, 2026 2:46:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 21, 2026 2:46:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 21, 2026 2:46:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 21, 2026 2:46:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 21, 2026 2:46:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 21, 2026 2:46:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 21, 2026 2:46:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 21, 2026 2:46:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 21, 2026 2:46:11 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 21, 2026 2:46:12 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260621 02:46:13.288334 9880 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.166.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.166.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.166.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/rangerkms_127.9.166.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.166.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/rangerkms_127.9.166.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.166.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.166.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.166.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/HTTP_127.9.166.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.166.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/HTTP_127.9.166.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 21 02:46:13 dist-test-slave-hvgn krb5kdc[20715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009973, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260621 02:46:13.455901 9880 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260621 02:46:13.478329 9880 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260621 02:46:14.251911 9880 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260621 02:46:14.252074 9880 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ranger-kms
I20260621 02:46:14.252198 9880 mini_ranger_kms.cc:227] Using postgres at 127.9.166.20:49667
2026-06-21 02:46:14,681 [I] DB FLAVOR :POSTGRES
2026-06-21 02:46:14,681 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 02:46:14,681 [I] Checking connection
2026-06-21 02:46:14,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-21 02:46:14,908 [I] connection success
2026-06-21 02:46:14,908 [I] --------- Verifying Ranger DB tables ---------
2026-06-21 02:46:14,908 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-21 02:46:14,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-21 02:46:15,137 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-21 02:46:15,138 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 02:46:15,138 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-21 02:46:15,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:49667/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-21 02:46:15,391 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260621 02:46:15.821138 9880 mini_ranger_kms.cc:310] Created kms service
I20260621 02:46:16.024165 9880 mini_ranger_kms.cc:326] Created kudu user
I20260621 02:46:16.068607 9880 mini_ranger_kms.cc:343] Created rangerkms user
Jun 21 02:46:16 dist-test-slave-hvgn krb5kdc[20715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009973, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
I20260621 02:46:16.474099 9880 mini_ranger_kms.cc:384] Added ranger policy
I20260621 02:46:16.474468 9880 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ranger-kms:/tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260621 02:46:16.474541 9880 mini_ranger_kms.cc:240] Using host: 127.9.166.20
I20260621 02:46:16.495229 9880 mini_ranger_kms.cc:276] Ranger KMS PID: 21108
I20260621 02:46:16.495482 9880 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.166.20:37693
Jun 21, 2026 2:46:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 2:46:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 2:46:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 21, 2026 2:46:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 21, 2026 2:46:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ranger-kms/ews/webapp].
Jun 21, 2026 2:46:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260621 02:46:18.810853 9880 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.355s user 0.142s sys 0.312s
I20260621 02:46:18.811141 9880 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260621 02:46:18.811378 9880 mini_ranger_kms.cc:401] 127.9.166.20:37693/kms/v1/keys
Jun 21 02:46:18 dist-test-slave-hvgn krb5kdc[20715](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782009978, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.166.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 21 02:46:19 dist-test-slave-hvgn krb5kdc[20715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009978, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.166.20@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
Jun 21 02:46:19 dist-test-slave-hvgn krb5kdc[20715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009978, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.166.20@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
Jun 21 02:46:21 dist-test-slave-hvgn krb5kdc[20715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:46:22.039170 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:43621
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:43621
--ranger_config_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.166.20:37693/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:46:22.372272 21159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:46:22.372604 21159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:46:22.372771 21159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:46:22.379967 21159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:46:22.380110 21159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:46:22.380191 21159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:46:22.380275 21159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:46:22.380347 21159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:46:22.390255 21159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.166.20:37693/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:43621
--ranger_config_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:43621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:46:22.391607 21159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:46:22.392930 21159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:46:22.405267 21164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:46:22.405239 21165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:46:22.405932 21159 server_base.cc:1034] running on GCE node
W20260621 02:46:22.406030 21167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:46:22.406837 21159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:46:22.408313 21159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:46:22.409561 21159 hybrid_clock.cc:648] HybridClock initialized: now 1782009982409504 us; error 48 us; skew 500 ppm
I20260621 02:46:22.409934 21159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:46:22 dist-test-slave-hvgn krb5kdc[20715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009982, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:46:22.413091 21159 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:46:22.414485 21159 webserver.cc:460] Webserver started at http://127.9.166.62:33905/ using document root <none> and password file <none>
I20260621 02:46:22.415134 21159 fs_manager.cc:359] Metadata directory not provided
I20260621 02:46:22.415269 21159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:46:22.415673 21159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:46:22 dist-test-slave-hvgn krb5kdc[20715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009982, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
I20260621 02:46:22.474434 21159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "c90de5db10f04bbb85e547567555a6f5"
format_stamp: "Formatted at 2026-06-21 02:46:22 on dist-test-slave-hvgn"
server_key: "40f7a486f6de3774aad7d567c54c6299"
server_key_iv: "010cfd045f02c0a33010ccf8ee552f66"
server_key_version: "kuduclusterkey@0"
I20260621 02:46:22.475538 21159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "c90de5db10f04bbb85e547567555a6f5"
format_stamp: "Formatted at 2026-06-21 02:46:22 on dist-test-slave-hvgn"
server_key: "40f7a486f6de3774aad7d567c54c6299"
server_key_iv: "010cfd045f02c0a33010ccf8ee552f66"
server_key_version: "kuduclusterkey@0"
I20260621 02:46:22.482532 21159 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260621 02:46:22.502200 21174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:46:22.503921 21159 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260621 02:46:22.504127 21159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "c90de5db10f04bbb85e547567555a6f5"
format_stamp: "Formatted at 2026-06-21 02:46:22 on dist-test-slave-hvgn"
server_key: "40f7a486f6de3774aad7d567c54c6299"
server_key_iv: "010cfd045f02c0a33010ccf8ee552f66"
server_key_version: "kuduclusterkey@0"
I20260621 02:46:22.504351 21159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:46:22.518342 21159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:46:22.522352 21159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:46:22.522646 21159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:46:22.540871 21159 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:43621
I20260621 02:46:22.540912 21225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:43621 every 8 connection(s)
I20260621 02:46:22.542181 21159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:46:22.545877 21226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:46:22.548180 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 21159
I20260621 02:46:22.548403 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:46:22.549188 9880 external_mini_cluster.cc:1428] Setting key 6add8eacdcf41d5e80fdff4def6648b3
I20260621 02:46:22.558569 21226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5: Bootstrap starting.
I20260621 02:46:22.562568 21226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5: Neither blocks nor log segments found. Creating new log.
Jun 21 02:46:22 dist-test-slave-hvgn krb5kdc[20715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:46:22.563823 21226 log.cc:824] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5: Log is configured to *not* fsync() on all Append() calls
I20260621 02:46:22.566754 21226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5: No bootstrap required, opened a new log
I20260621 02:46:22.571919 21229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:22.551228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.9.166.62:43621)
0621 02:46:22.552123 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0621 02:46:22.552133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:46:22.552176 (+ 43us) server_negotiation.cc:407] Connection header received
0621 02:46:22.553781 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:46:22.553803 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:46:22.554193 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:46:22.554505 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:46:22.556167 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:22.557479 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:22.559189 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:22.559932 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:22.560070 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:22.564664 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:46:22.564673 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:46:22.564687 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:46:22.564725 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:46:22.567924 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:46:22.568614 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:46:22.568626 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:46:22.568633 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:46:22.568784 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:46:22.569178 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:46:22.569189 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:46:22.569197 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:46:22.569696 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:46:22.570009 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0621 02:46:22.570455 (+ 446us) server_negotiation.cc:299] Negotiation successful
0621 02:46:22.570849 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":441,"threads_started":1}
I20260621 02:46:22.574741 21226 raft_consensus.cc:348] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c90de5db10f04bbb85e547567555a6f5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 43621 } }
I20260621 02:46:22.575163 21226 raft_consensus.cc:374] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:46:22.575273 21226 raft_consensus.cc:729] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c90de5db10f04bbb85e547567555a6f5, State: Initialized, Role: FOLLOWER
I20260621 02:46:22.575827 21226 consensus_queue.cc:260] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c90de5db10f04bbb85e547567555a6f5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 43621 } }
I20260621 02:46:22.576068 21226 raft_consensus.cc:388] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:46:22.576207 21226 raft_consensus.cc:482] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:46:22.576377 21226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:46:22.578482 21226 raft_consensus.cc:504] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c90de5db10f04bbb85e547567555a6f5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 43621 } }
I20260621 02:46:22.578966 21226 leader_election.cc:302] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c90de5db10f04bbb85e547567555a6f5; no voters:
I20260621 02:46:22.579622 21226 leader_election.cc:288] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:46:22.579756 21231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:46:22.581113 21231 raft_consensus.cc:686] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5 [term 1 LEADER]: Becoming Leader. State: Replica: c90de5db10f04bbb85e547567555a6f5, State: Running, Role: LEADER
I20260621 02:46:22.581789 21226 sys_catalog.cc:564] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:46:22.581664 21231 consensus_queue.cc:237] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c90de5db10f04bbb85e547567555a6f5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 43621 } }
I20260621 02:46:22.584420 21233 sys_catalog.cc:455] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c90de5db10f04bbb85e547567555a6f5. Latest consensus state: current_term: 1 leader_uuid: "c90de5db10f04bbb85e547567555a6f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c90de5db10f04bbb85e547567555a6f5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 43621 } } }
I20260621 02:46:22.584808 21233 sys_catalog.cc:458] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5 [sys.catalog]: This master's current role is: LEADER
I20260621 02:46:22.584779 21232 sys_catalog.cc:455] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c90de5db10f04bbb85e547567555a6f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c90de5db10f04bbb85e547567555a6f5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 43621 } } }
I20260621 02:46:22.585144 21232 sys_catalog.cc:458] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5 [sys.catalog]: This master's current role is: LEADER
I20260621 02:46:22.586750 21240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:46:22.591085 21240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:46:22.591840 21226 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 02:46:22.608873 21240 catalog_manager.cc:1347] Generated new cluster ID: d941c35046a04f4b8231949f5379a76c
I20260621 02:46:22.609103 21240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:46:22.669557 21240 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:46:22.671563 21240 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:46:22.684363 21240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5: Generated new TSK 0
I20260621 02:46:22.685546 21240 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
I20260621 02:46:23.385426 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.166.20:37693/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:43621
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:46:23 dist-test-slave-hvgn krb5kdc[20715](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782009983, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260621 02:46:23.799057 21271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:46:23.799459 21271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:46:23.799620 21271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:46:23.811187 21271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:46:23.811335 21271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:46:23.811582 21271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:46:23.827003 21271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.166.20:37693/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:43621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:46:23.828812 21271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:46:23.830581 21271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:46:23.847180 21282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:46:23.847272 21284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:46:23.847862 21281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:46:23.848784 21271 server_base.cc:1034] running on GCE node
I20260621 02:46:23.849444 21271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:46:23.850473 21271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:46:23.851706 21271 hybrid_clock.cc:648] HybridClock initialized: now 1782009983851640 us; error 61 us; skew 500 ppm
I20260621 02:46:23.852020 21271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:46:23 dist-test-slave-hvgn krb5kdc[20715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009983, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:46:23.855842 21271 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:46:23.857496 21271 webserver.cc:460] Webserver started at http://127.9.166.1:42033/ using document root <none> and password file <none>
I20260621 02:46:23.858184 21271 fs_manager.cc:359] Metadata directory not provided
I20260621 02:46:23.858325 21271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:46:23.858633 21271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:46:23 dist-test-slave-hvgn krb5kdc[20715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009983, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
I20260621 02:46:23.877499 21271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "c527fa4543344ba1b99f1b84aaa76529"
format_stamp: "Formatted at 2026-06-21 02:46:23 on dist-test-slave-hvgn"
server_key: "cb620b73e2f4cea7aa45034a1168e54c"
server_key_iv: "c07de621deb7aca21069ba0d83c317d1"
server_key_version: "kuduclusterkey@0"
I20260621 02:46:23.878710 21271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "c527fa4543344ba1b99f1b84aaa76529"
format_stamp: "Formatted at 2026-06-21 02:46:23 on dist-test-slave-hvgn"
server_key: "cb620b73e2f4cea7aa45034a1168e54c"
server_key_iv: "c07de621deb7aca21069ba0d83c317d1"
server_key_version: "kuduclusterkey@0"
I20260621 02:46:23.901278 21271 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.006s sys 0.011s
I20260621 02:46:23.928857 21291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:46:23.930537 21271 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.001s
I20260621 02:46:23.930732 21271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "c527fa4543344ba1b99f1b84aaa76529"
format_stamp: "Formatted at 2026-06-21 02:46:23 on dist-test-slave-hvgn"
server_key: "cb620b73e2f4cea7aa45034a1168e54c"
server_key_iv: "c07de621deb7aca21069ba0d83c317d1"
server_key_version: "kuduclusterkey@0"
I20260621 02:46:23.930966 21271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:46:23.962173 21271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:46:23.967454 21271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:46:23.967864 21271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:46:23.969208 21271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:46:23.976329 21271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:46:23.976501 21271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260621 02:46:23.976701 21271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:46:23.976840 21271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:46:24.025560 21271 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:36341
I20260621 02:46:24.025681 21403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:36341 every 8 connection(s)
I20260621 02:46:24.026955 21271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:46:24.034025 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 21271
I20260621 02:46:24.034263 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:46:24.035125 9880 external_mini_cluster.cc:1428] Setting key e1482159c8dee48d806f29603b42cf66
Jun 21 02:46:24 dist-test-slave-hvgn krb5kdc[20715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009983, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.2@KRBTEST.COM" created.
I20260621 02:46:24.071192 21406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:24.029855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:59227 (local address 127.9.166.62:43621)
0621 02:46:24.030330 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0621 02:46:24.030336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:46:24.031460 (+ 1124us) server_negotiation.cc:407] Connection header received
0621 02:46:24.033225 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:46:24.033233 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:46:24.033371 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:46:24.033524 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:46:24.035529 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:24.036555 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:24.039157 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:24.039956 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:24.040091 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:24.063350 (+ 23259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:46:24.063359 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:46:24.063365 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:46:24.063440 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:46:24.067168 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:46:24.068256 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:46:24.068263 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:46:24.068269 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:46:24.068533 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:46:24.069077 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:46:24.069082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:46:24.069086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:46:24.069446 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:46:24.069572 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0621 02:46:24.070562 (+ 990us) server_negotiation.cc:299] Negotiation successful
0621 02:46:24.070799 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":230,"threads_started":1}
I20260621 02:46:24.073763 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:24.030450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:43621 (local address 127.9.166.1:59227)
0621 02:46:24.031273 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0621 02:46:24.031319 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 02:46:24.032950 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:46:24.033787 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:46:24.033799 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:46:24.034437 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:46:24.035281 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:46:24.035309 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:46:24.036725 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:46:24.036733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:46:24.038973 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:46:24.038997 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:46:24.040394 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:46:24.040403 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:46:24.040744 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:24.055730 (+ 14986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:46:24.055770 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:46:24.060219 (+ 4449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:46:24.067437 (+ 7218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:46:24.067448 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:46:24.067468 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:46:24.068095 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:46:24.068691 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:46:24.068696 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:46:24.068700 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:46:24.068941 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:46:24.069666 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:46:24.069675 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:46:24.070111 (+ 436us) client_negotiation.cc:769] Sending connection context
0621 02:46:24.071757 (+ 1646us) client_negotiation.cc:240] Negotiation successful
0621 02:46:24.072328 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":348,"threads_started":1}
I20260621 02:46:24.075960 21404 heartbeater.cc:347] Connected to a master server at 127.9.166.62:43621
I20260621 02:46:24.076311 21404 heartbeater.cc:464] Registering TS with master...
I20260621 02:46:24.077148 21404 heartbeater.cc:511] Master 127.9.166.62:43621 requested a full tablet report, sending...
I20260621 02:46:24.079666 21191 ts_manager.cc:194] Registered new tserver with Master: c527fa4543344ba1b99f1b84aaa76529 (127.9.166.1:36341)
I20260621 02:46:24.081897 21191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:59227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
I20260621 02:46:24.100828 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.166.20:37693/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:43621
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:46:24 dist-test-slave-hvgn krb5kdc[20715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009983, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
Jun 21 02:46:24 dist-test-slave-hvgn krb5kdc[20715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009983, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
W20260621 02:46:24.591482 21410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:46:24.591988 21410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:46:24.592159 21410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:46:24.603096 21410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:46:24.603308 21410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:46:24.603536 21410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:46:24.617753 21410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.166.20:37693/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:43621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:46:24.619907 21410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:46:24.621588 21410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:46:24.676098 21418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:46:24.676839 21417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:46:24.686480 21410 server_base.cc:1034] running on GCE node
W20260621 02:46:24.687518 21420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:46:24.688409 21410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:46:24.709069 21410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:46:24.712548 21410 hybrid_clock.cc:648] HybridClock initialized: now 1782009984712427 us; error 120 us; skew 500 ppm
I20260621 02:46:24.713088 21410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:46:24 dist-test-slave-hvgn krb5kdc[20715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009984, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:46:24.720777 21410 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:46:24.738574 21410 webserver.cc:460] Webserver started at http://127.9.166.2:45041/ using document root <none> and password file <none>
I20260621 02:46:24.739763 21410 fs_manager.cc:359] Metadata directory not provided
I20260621 02:46:24.740006 21410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:46:24.740417 21410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:46:24 dist-test-slave-hvgn krb5kdc[20715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009984, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
I20260621 02:46:24.767122 21410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "0fb1ebcccf4248a49808a9e6b4b5ee08"
format_stamp: "Formatted at 2026-06-21 02:46:24 on dist-test-slave-hvgn"
server_key: "1ad8f146610f918a46e7c7fcb4b35a5a"
server_key_iv: "aff2846e65f53e8fedc36d252dc0133e"
server_key_version: "kuduclusterkey@0"
I20260621 02:46:24.768289 21410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "0fb1ebcccf4248a49808a9e6b4b5ee08"
format_stamp: "Formatted at 2026-06-21 02:46:24 on dist-test-slave-hvgn"
server_key: "1ad8f146610f918a46e7c7fcb4b35a5a"
server_key_iv: "aff2846e65f53e8fedc36d252dc0133e"
server_key_version: "kuduclusterkey@0"
I20260621 02:46:24.781913 21410 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.000s sys 0.011s
I20260621 02:46:24.820061 21427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:46:24.822201 21410 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.004s
I20260621 02:46:24.822510 21410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "0fb1ebcccf4248a49808a9e6b4b5ee08"
format_stamp: "Formatted at 2026-06-21 02:46:24 on dist-test-slave-hvgn"
server_key: "1ad8f146610f918a46e7c7fcb4b35a5a"
server_key_iv: "aff2846e65f53e8fedc36d252dc0133e"
server_key_version: "kuduclusterkey@0"
I20260621 02:46:24.822875 21410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:46:24.865902 21410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:46:24.871121 21410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:46:24.871496 21410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:46:24.872637 21410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:46:24.874353 21410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:46:24.874456 21410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:46:24.874584 21410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:46:24.874653 21410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:46:24.944363 21410 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:37815
I20260621 02:46:24.944388 21546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:37815 every 8 connection(s)
I20260621 02:46:24.945705 21410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:46:24.952023 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 21410
I20260621 02:46:24.952196 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:46:24.952764 9880 external_mini_cluster.cc:1428] Setting key 30f2db6c4b25bba06ccdedd69e997070
Jun 21 02:46:24 dist-test-slave-hvgn krb5kdc[20715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009984, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:46:24.969061 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:24.948601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:51681 (local address 127.9.166.62:43621)
0621 02:46:24.949175 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0621 02:46:24.949182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:46:24.950227 (+ 1045us) server_negotiation.cc:407] Connection header received
0621 02:46:24.952079 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:46:24.952084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:46:24.952175 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:46:24.952291 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:46:24.953929 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:24.955144 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:24.957766 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:24.958405 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:24.958541 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:24.963060 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:46:24.963068 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:46:24.963074 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:46:24.963119 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:46:24.965826 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:46:24.966661 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:46:24.966667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:46:24.966670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:46:24.966799 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:46:24.967233 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:46:24.967238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:46:24.967242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:46:24.967628 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:46:24.967784 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 02:46:24.968460 (+ 676us) server_negotiation.cc:299] Negotiation successful
0621 02:46:24.968685 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":315,"threads_started":1}
I20260621 02:46:24.970189 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:24.949224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:43621 (local address 127.9.166.2:51681)
0621 02:46:24.950050 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0621 02:46:24.950096 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 02:46:24.951819 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:46:24.952557 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:46:24.952565 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:46:24.953050 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:46:24.953755 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:46:24.953775 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:46:24.955326 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:46:24.955332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:46:24.957604 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:46:24.957621 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:46:24.958660 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:46:24.958669 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:46:24.959004 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:24.959874 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:46:24.959910 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:46:24.962813 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:46:24.965983 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:46:24.965992 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:46:24.966008 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:46:24.966537 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:46:24.966917 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:46:24.966921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:46:24.966925 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:46:24.967115 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:46:24.967744 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:46:24.967752 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:46:24.968090 (+ 338us) client_negotiation.cc:769] Sending connection context
0621 02:46:24.968343 (+ 253us) client_negotiation.cc:240] Negotiation successful
0621 02:46:24.968846 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":373,"threads_started":1}
I20260621 02:46:24.971642 21547 heartbeater.cc:347] Connected to a master server at 127.9.166.62:43621
I20260621 02:46:24.971966 21547 heartbeater.cc:464] Registering TS with master...
I20260621 02:46:24.972640 21547 heartbeater.cc:511] Master 127.9.166.62:43621 requested a full tablet report, sending...
I20260621 02:46:24.974100 21191 ts_manager.cc:194] Registered new tserver with Master: 0fb1ebcccf4248a49808a9e6b4b5ee08 (127.9.166.2:37815)
I20260621 02:46:24.974905 21191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:51681
WARNING: no policy specified for kudu/127.9.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
I20260621 02:46:25.014284 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.166.20:37693/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:43621
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:46:25.085877 21404 heartbeater.cc:503] Master 127.9.166.62:43621 was elected leader, sending a full tablet report...
W20260621 02:46:25.359802 21553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:46:25.360066 21553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:46:25.360227 21553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:46:25.367221 21553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:46:25.367328 21553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:46:25.367532 21553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:46:25.376854 21553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.166.20:37693/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:43621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:46:25.378155 21553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:46:25.379475 21553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:46:25.392318 21561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:46:25.392279 21559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:46:25.392395 21558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:46:25.393214 21553 server_base.cc:1034] running on GCE node
I20260621 02:46:25.393749 21553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:46:25.394644 21553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:46:25.395886 21553 hybrid_clock.cc:648] HybridClock initialized: now 1782009985395849 us; error 49 us; skew 500 ppm
I20260621 02:46:25.396193 21553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:46:25 dist-test-slave-hvgn krb5kdc[20715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009985, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:46:25.399255 21553 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:46:25.400729 21553 webserver.cc:460] Webserver started at http://127.9.166.3:37793/ using document root <none> and password file <none>
I20260621 02:46:25.401371 21553 fs_manager.cc:359] Metadata directory not provided
I20260621 02:46:25.401517 21553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:46:25.401818 21553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:46:25 dist-test-slave-hvgn krb5kdc[20715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009985, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
I20260621 02:46:25.417475 21553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "f29bee8cc0374f7283a28d26b64309db"
format_stamp: "Formatted at 2026-06-21 02:46:25 on dist-test-slave-hvgn"
server_key: "b05fdc49704aa54b80b2ad0d66f2dea2"
server_key_iv: "e50eab91fada6fcc38231e3be38a874d"
server_key_version: "kuduclusterkey@0"
I20260621 02:46:25.418486 21553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "f29bee8cc0374f7283a28d26b64309db"
format_stamp: "Formatted at 2026-06-21 02:46:25 on dist-test-slave-hvgn"
server_key: "b05fdc49704aa54b80b2ad0d66f2dea2"
server_key_iv: "e50eab91fada6fcc38231e3be38a874d"
server_key_version: "kuduclusterkey@0"
I20260621 02:46:25.424083 21553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260621 02:46:25.438906 21568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:46:25.440557 21553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:46:25.440724 21553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "f29bee8cc0374f7283a28d26b64309db"
format_stamp: "Formatted at 2026-06-21 02:46:25 on dist-test-slave-hvgn"
server_key: "b05fdc49704aa54b80b2ad0d66f2dea2"
server_key_iv: "e50eab91fada6fcc38231e3be38a874d"
server_key_version: "kuduclusterkey@0"
I20260621 02:46:25.440944 21553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:46:25.456004 21553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:46:25.459951 21553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:46:25.460237 21553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:46:25.461133 21553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:46:25.462445 21553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:46:25.462515 21553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:46:25.462670 21553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:46:25.462742 21553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:46:25.497267 21553 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:40429
I20260621 02:46:25.497287 21680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:40429 every 8 connection(s)
I20260621 02:46:25.498692 21553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:46:25.499229 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 21553
I20260621 02:46:25.499452 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:46:25.500074 9880 external_mini_cluster.cc:1428] Setting key 9a75f6635a608f61aa9887274cd8f488
Jun 21 02:46:25 dist-test-slave-hvgn krb5kdc[20715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009985, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:46:25.519722 21683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:25.501481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:35589 (local address 127.9.166.62:43621)
0621 02:46:25.501963 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0621 02:46:25.501968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:46:25.503125 (+ 1157us) server_negotiation.cc:407] Connection header received
0621 02:46:25.504936 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:46:25.504940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:46:25.505020 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:46:25.505118 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:46:25.506446 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.507225 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.509241 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.509841 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.509943 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:25.513300 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:46:25.513306 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:46:25.513309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:46:25.513338 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:46:25.516669 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:46:25.517376 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:46:25.517380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:46:25.517382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:46:25.517505 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:46:25.517891 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:46:25.517894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:46:25.517896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:46:25.518151 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:46:25.518256 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 02:46:25.518893 (+ 637us) server_negotiation.cc:299] Negotiation successful
0621 02:46:25.519043 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":235,"threads_started":1}
I20260621 02:46:25.520634 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:25.502166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:43621 (local address 127.9.166.3:35589)
0621 02:46:25.502970 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0621 02:46:25.503015 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:46:25.504693 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:46:25.505319 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:46:25.505327 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:46:25.505722 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:46:25.506299 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:46:25.506316 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.507351 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.507354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:46:25.509108 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:46:25.509120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.510050 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.510055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:46:25.510256 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:25.510897 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:46:25.510920 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:46:25.513066 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:46:25.516789 (+ 3723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:46:25.516796 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:46:25.516809 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:46:25.517256 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:46:25.517596 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:46:25.517599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:46:25.517601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:46:25.517786 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:46:25.518251 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:46:25.518257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:46:25.518552 (+ 295us) client_negotiation.cc:769] Sending connection context
0621 02:46:25.518771 (+ 219us) client_negotiation.cc:240] Negotiation successful
0621 02:46:25.519139 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":354,"threads_started":1}
I20260621 02:46:25.522006 21681 heartbeater.cc:347] Connected to a master server at 127.9.166.62:43621
I20260621 02:46:25.522293 21681 heartbeater.cc:464] Registering TS with master...
I20260621 02:46:25.523015 21681 heartbeater.cc:511] Master 127.9.166.62:43621 requested a full tablet report, sending...
I20260621 02:46:25.524473 21191 ts_manager.cc:194] Registered new tserver with Master: f29bee8cc0374f7283a28d26b64309db (127.9.166.3:40429)
I20260621 02:46:25.525384 21191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:35589
I20260621 02:46:25.528287 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:46:25.545300 21683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:25.532684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.9.166.62:43621)
0621 02:46:25.532829 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0621 02:46:25.532833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:46:25.533328 (+ 495us) server_negotiation.cc:407] Connection header received
0621 02:46:25.533449 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:46:25.533453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:46:25.533520 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:46:25.533596 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:46:25.535028 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.535865 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.537180 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.537594 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.537682 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:25.539937 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:46:25.539942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:46:25.539945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:46:25.539973 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:46:25.542618 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:46:25.543290 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:46:25.543300 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:46:25.543304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:46:25.543510 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:46:25.543892 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:46:25.543900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:46:25.543905 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:46:25.544188 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:46:25.544431 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0621 02:46:25.544751 (+ 320us) server_negotiation.cc:299] Negotiation successful
0621 02:46:25.544967 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 02:46:25.549206 21191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:46:25.556149 21191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:46:25.580376 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:25.569112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:36341 (local address 127.0.0.1:44362)
0621 02:46:25.570421 (+ 1309us) negotiation.cc:107] Waiting for socket to connect
0621 02:46:25.570510 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0621 02:46:25.570677 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:46:25.572021 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:46:25.572028 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:46:25.572056 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:46:25.572505 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:46:25.572522 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.574774 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.574779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:46:25.577454 (+ 2675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:46:25.577468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.579460 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.579467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:46:25.579649 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:25.579729 (+ 80us) client_negotiation.cc:769] Sending connection context
0621 02:46:25.579841 (+ 112us) client_negotiation.cc:240] Negotiation successful
0621 02:46:25.580025 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1096,"thread_start_us":881,"threads_started":1}
I20260621 02:46:25.581926 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:25.569358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.9.166.1:36341)
0621 02:46:25.571410 (+ 2052us) server_negotiation.cc:206] Beginning negotiation
0621 02:46:25.571418 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:46:25.571442 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:46:25.571534 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:46:25.571541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:46:25.571760 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:46:25.572713 (+ 953us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:46:25.572957 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.574621 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.577647 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.579210 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.580989 (+ 1779us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:25.581214 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0621 02:46:25.581287 (+ 73us) server_negotiation.cc:299] Negotiation successful
0621 02:46:25.581601 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1855,"thread_start_us":1627,"threads_started":1}
I20260621 02:46:25.582190 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:25.568605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33332 (local address 127.9.166.2:37815)
0621 02:46:25.569771 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0621 02:46:25.569780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:46:25.570616 (+ 836us) server_negotiation.cc:407] Connection header received
0621 02:46:25.570918 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:46:25.570930 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:46:25.571196 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:46:25.572380 (+ 1184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:46:25.573304 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.574764 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.578175 (+ 3411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.580602 (+ 2427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.580697 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:25.580905 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0621 02:46:25.581652 (+ 747us) server_negotiation.cc:299] Negotiation successful
0621 02:46:25.581856 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1010,"thread_start_us":926,"threads_started":1}
I20260621 02:46:25.582840 21696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:25.568596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:37815 (local address 127.0.0.1:33332)
0621 02:46:25.570421 (+ 1825us) negotiation.cc:107] Waiting for socket to connect
0621 02:46:25.570510 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0621 02:46:25.570693 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:46:25.572668 (+ 1975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:46:25.572671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:46:25.572694 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:46:25.573109 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:46:25.573122 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.575919 (+ 2797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.575923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:46:25.578004 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:46:25.578018 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.581097 (+ 3079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.581103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:46:25.581430 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:25.581471 (+ 41us) client_negotiation.cc:769] Sending connection context
0621 02:46:25.582380 (+ 909us) client_negotiation.cc:240] Negotiation successful
0621 02:46:25.582507 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1555,"thread_start_us":1408,"threads_started":1}
I20260621 02:46:25.586139 21339 tablet_service.cc:1467] Processing CreateTablet for tablet e4ee3e63fcd542be9b98c988af25317b (DEFAULT_TABLE table=test-table [id=43107d8092b146afb0a4e3f914f0434a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:46:25.586503 21482 tablet_service.cc:1467] Processing CreateTablet for tablet e4ee3e63fcd542be9b98c988af25317b (DEFAULT_TABLE table=test-table [id=43107d8092b146afb0a4e3f914f0434a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:46:25.587744 21339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ee3e63fcd542be9b98c988af25317b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:46:25.587764 21482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ee3e63fcd542be9b98c988af25317b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:46:25.588506 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:25.570869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:40429 (local address 127.0.0.1:59908)
0621 02:46:25.580715 (+ 9846us) negotiation.cc:107] Waiting for socket to connect
0621 02:46:25.580734 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:46:25.580827 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:46:25.582723 (+ 1896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:46:25.582728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:46:25.582758 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:46:25.582972 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:46:25.582982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.584904 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.584912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:46:25.586232 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:46:25.586249 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.587789 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.587795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:46:25.587984 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:25.588018 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 02:46:25.588107 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 02:46:25.588234 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9755,"thread_start_us":10434,"threads_started":1}
I20260621 02:46:25.588631 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:25.571150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.9.166.3:40429)
0621 02:46:25.582079 (+ 10929us) server_negotiation.cc:206] Beginning negotiation
0621 02:46:25.582086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:46:25.582119 (+ 33us) server_negotiation.cc:407] Connection header received
0621 02:46:25.582205 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:46:25.582212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:46:25.582548 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:46:25.582795 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:46:25.583266 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.584665 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.586418 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.587639 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.587777 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:25.587994 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0621 02:46:25.588141 (+ 147us) server_negotiation.cc:299] Negotiation successful
0621 02:46:25.588294 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10722,"thread_start_us":10540,"threads_started":1}
I20260621 02:46:25.591741 21616 tablet_service.cc:1467] Processing CreateTablet for tablet e4ee3e63fcd542be9b98c988af25317b (DEFAULT_TABLE table=test-table [id=43107d8092b146afb0a4e3f914f0434a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:46:25.592841 21616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ee3e63fcd542be9b98c988af25317b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:46:25.602157 21701 tablet_bootstrap.cc:492] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08: Bootstrap starting.
I20260621 02:46:25.602892 21700 tablet_bootstrap.cc:492] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529: Bootstrap starting.
I20260621 02:46:25.604496 21702 tablet_bootstrap.cc:492] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db: Bootstrap starting.
I20260621 02:46:25.607182 21700 tablet_bootstrap.cc:654] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529: Neither blocks nor log segments found. Creating new log.
I20260621 02:46:25.607465 21701 tablet_bootstrap.cc:654] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08: Neither blocks nor log segments found. Creating new log.
I20260621 02:46:25.607934 21702 tablet_bootstrap.cc:654] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db: Neither blocks nor log segments found. Creating new log.
I20260621 02:46:25.608525 21700 log.cc:824] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529: Log is configured to *not* fsync() on all Append() calls
I20260621 02:46:25.608556 21701 log.cc:824] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08: Log is configured to *not* fsync() on all Append() calls
I20260621 02:46:25.609112 21702 log.cc:824] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db: Log is configured to *not* fsync() on all Append() calls
I20260621 02:46:25.612249 21701 tablet_bootstrap.cc:492] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08: No bootstrap required, opened a new log
I20260621 02:46:25.612475 21702 tablet_bootstrap.cc:492] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db: No bootstrap required, opened a new log
I20260621 02:46:25.612514 21701 ts_tablet_manager.cc:1397] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260621 02:46:25.612588 21700 tablet_bootstrap.cc:492] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529: No bootstrap required, opened a new log
I20260621 02:46:25.612737 21702 ts_tablet_manager.cc:1397] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 02:46:25.612875 21700 ts_tablet_manager.cc:1397] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260621 02:46:25.619913 21701 raft_consensus.cc:348] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fb1ebcccf4248a49808a9e6b4b5ee08" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37815 } } peers { permanent_uuid: "f29bee8cc0374f7283a28d26b64309db" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40429 } } peers { permanent_uuid: "c527fa4543344ba1b99f1b84aaa76529" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36341 } }
I20260621 02:46:25.620266 21701 raft_consensus.cc:374] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:46:25.620398 21701 raft_consensus.cc:729] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fb1ebcccf4248a49808a9e6b4b5ee08, State: Initialized, Role: FOLLOWER
I20260621 02:46:25.620980 21701 consensus_queue.cc:260] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fb1ebcccf4248a49808a9e6b4b5ee08" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37815 } } peers { permanent_uuid: "f29bee8cc0374f7283a28d26b64309db" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40429 } } peers { permanent_uuid: "c527fa4543344ba1b99f1b84aaa76529" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36341 } }
I20260621 02:46:25.621999 21547 heartbeater.cc:503] Master 127.9.166.62:43621 was elected leader, sending a full tablet report...
I20260621 02:46:25.622691 21701 ts_tablet_manager.cc:1428] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260621 02:46:25.623286 21700 raft_consensus.cc:348] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fb1ebcccf4248a49808a9e6b4b5ee08" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37815 } } peers { permanent_uuid: "f29bee8cc0374f7283a28d26b64309db" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40429 } } peers { permanent_uuid: "c527fa4543344ba1b99f1b84aaa76529" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36341 } }
I20260621 02:46:25.623286 21702 raft_consensus.cc:348] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fb1ebcccf4248a49808a9e6b4b5ee08" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37815 } } peers { permanent_uuid: "f29bee8cc0374f7283a28d26b64309db" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40429 } } peers { permanent_uuid: "c527fa4543344ba1b99f1b84aaa76529" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36341 } }
I20260621 02:46:25.623716 21700 raft_consensus.cc:374] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:46:25.623705 21702 raft_consensus.cc:374] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:46:25.623811 21702 raft_consensus.cc:729] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29bee8cc0374f7283a28d26b64309db, State: Initialized, Role: FOLLOWER
I20260621 02:46:25.623811 21700 raft_consensus.cc:729] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c527fa4543344ba1b99f1b84aaa76529, State: Initialized, Role: FOLLOWER
I20260621 02:46:25.624388 21702 consensus_queue.cc:260] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fb1ebcccf4248a49808a9e6b4b5ee08" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37815 } } peers { permanent_uuid: "f29bee8cc0374f7283a28d26b64309db" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40429 } } peers { permanent_uuid: "c527fa4543344ba1b99f1b84aaa76529" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36341 } }
I20260621 02:46:25.624387 21700 consensus_queue.cc:260] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fb1ebcccf4248a49808a9e6b4b5ee08" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37815 } } peers { permanent_uuid: "f29bee8cc0374f7283a28d26b64309db" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40429 } } peers { permanent_uuid: "c527fa4543344ba1b99f1b84aaa76529" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36341 } }
I20260621 02:46:25.625990 21681 heartbeater.cc:503] Master 127.9.166.62:43621 was elected leader, sending a full tablet report...
I20260621 02:46:25.626101 21702 ts_tablet_manager.cc:1428] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260621 02:46:25.626418 21700 ts_tablet_manager.cc:1428] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260621 02:46:25.697767 21548 tablet.cc:2369] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:46:25.750581 21682 tablet.cc:2369] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:46:25.780328 21405 tablet.cc:2369] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:46:25.814390 21706 raft_consensus.cc:482] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:46:25.814749 21706 raft_consensus.cc:504] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fb1ebcccf4248a49808a9e6b4b5ee08" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37815 } } peers { permanent_uuid: "f29bee8cc0374f7283a28d26b64309db" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40429 } } peers { permanent_uuid: "c527fa4543344ba1b99f1b84aaa76529" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36341 } }
I20260621 02:46:25.816530 21706 leader_election.cc:288] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f29bee8cc0374f7283a28d26b64309db (127.9.166.3:40429), c527fa4543344ba1b99f1b84aaa76529 (127.9.166.1:36341)
I20260621 02:46:25.826012 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:25.817084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:56829 (local address 127.9.166.1:36341)
0621 02:46:25.817285 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0621 02:46:25.817292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:46:25.818033 (+ 741us) server_negotiation.cc:407] Connection header received
0621 02:46:25.818241 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:46:25.818246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:46:25.818317 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:46:25.818421 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:46:25.819302 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.820464 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.823375 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.824676 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.824820 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:25.824918 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0621 02:46:25.825605 (+ 687us) server_negotiation.cc:299] Negotiation successful
0621 02:46:25.825789 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:46:25.826006 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:25.817079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:55113 (local address 127.9.166.3:40429)
0621 02:46:25.817289 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 02:46:25.817294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:46:25.818030 (+ 736us) server_negotiation.cc:407] Connection header received
0621 02:46:25.818235 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:46:25.818240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:46:25.818317 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:46:25.818421 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:46:25.819300 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.820451 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.823375 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.824684 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.824813 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:25.824917 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 02:46:25.825605 (+ 688us) server_negotiation.cc:299] Negotiation successful
0621 02:46:25.825746 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:46:25.826251 21710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:25.816914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:36341 (local address 127.9.166.2:56829)
0621 02:46:25.817895 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0621 02:46:25.817916 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:46:25.818112 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:46:25.818457 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:46:25.818462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:46:25.818511 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:46:25.819144 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:46:25.819152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.820601 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.820607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:46:25.823172 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:46:25.823187 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.824979 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.824986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:46:25.825300 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:25.825431 (+ 131us) client_negotiation.cc:769] Sending connection context
0621 02:46:25.825528 (+ 97us) client_negotiation.cc:240] Negotiation successful
0621 02:46:25.825754 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":275,"spinlock_wait_cycles":46336,"thread_start_us":625,"threads_started":1}
I20260621 02:46:25.826273 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:25.816866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:40429 (local address 127.9.166.2:55113)
0621 02:46:25.817895 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0621 02:46:25.817916 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:46:25.818070 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:46:25.818457 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:46:25.818462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:46:25.818511 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:46:25.819137 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:46:25.819149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.820602 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.820607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:46:25.823172 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:46:25.823187 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.824979 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.824986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:46:25.825296 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:25.825431 (+ 135us) client_negotiation.cc:769] Sending connection context
0621 02:46:25.825528 (+ 97us) client_negotiation.cc:240] Negotiation successful
0621 02:46:25.825754 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"mutex_wait_us":367,"spinlock_wait_cycles":2048,"thread_start_us":635,"threads_started":1}
I20260621 02:46:25.827464 21359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ee3e63fcd542be9b98c988af25317b" candidate_uuid: "0fb1ebcccf4248a49808a9e6b4b5ee08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c527fa4543344ba1b99f1b84aaa76529" is_pre_election: true
I20260621 02:46:25.827508 21636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ee3e63fcd542be9b98c988af25317b" candidate_uuid: "0fb1ebcccf4248a49808a9e6b4b5ee08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29bee8cc0374f7283a28d26b64309db" is_pre_election: true
I20260621 02:46:25.827975 21359 raft_consensus.cc:2454] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fb1ebcccf4248a49808a9e6b4b5ee08 in term 0.
I20260621 02:46:25.828035 21636 raft_consensus.cc:2454] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fb1ebcccf4248a49808a9e6b4b5ee08 in term 0.
I20260621 02:46:25.828632 21428 leader_election.cc:302] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fb1ebcccf4248a49808a9e6b4b5ee08, c527fa4543344ba1b99f1b84aaa76529; no voters:
I20260621 02:46:25.829056 21706 raft_consensus.cc:2781] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:46:25.829197 21706 raft_consensus.cc:482] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:46:25.829298 21706 raft_consensus.cc:3037] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:46:25.831605 21706 raft_consensus.cc:504] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fb1ebcccf4248a49808a9e6b4b5ee08" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37815 } } peers { permanent_uuid: "f29bee8cc0374f7283a28d26b64309db" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40429 } } peers { permanent_uuid: "c527fa4543344ba1b99f1b84aaa76529" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36341 } }
I20260621 02:46:25.832185 21706 leader_election.cc:288] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [CANDIDATE]: Term 1 election: Requested vote from peers f29bee8cc0374f7283a28d26b64309db (127.9.166.3:40429), c527fa4543344ba1b99f1b84aaa76529 (127.9.166.1:36341)
I20260621 02:46:25.832590 21636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ee3e63fcd542be9b98c988af25317b" candidate_uuid: "0fb1ebcccf4248a49808a9e6b4b5ee08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29bee8cc0374f7283a28d26b64309db"
I20260621 02:46:25.832612 21359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ee3e63fcd542be9b98c988af25317b" candidate_uuid: "0fb1ebcccf4248a49808a9e6b4b5ee08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c527fa4543344ba1b99f1b84aaa76529"
I20260621 02:46:25.832782 21636 raft_consensus.cc:3037] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:46:25.832792 21359 raft_consensus.cc:3037] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:46:25.834935 21636 raft_consensus.cc:2454] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fb1ebcccf4248a49808a9e6b4b5ee08 in term 1.
I20260621 02:46:25.835105 21359 raft_consensus.cc:2454] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fb1ebcccf4248a49808a9e6b4b5ee08 in term 1.
I20260621 02:46:25.835355 21431 leader_election.cc:302] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fb1ebcccf4248a49808a9e6b4b5ee08, f29bee8cc0374f7283a28d26b64309db; no voters:
I20260621 02:46:25.835676 21706 raft_consensus.cc:2781] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:46:25.836089 21706 raft_consensus.cc:686] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [term 1 LEADER]: Becoming Leader. State: Replica: 0fb1ebcccf4248a49808a9e6b4b5ee08, State: Running, Role: LEADER
I20260621 02:46:25.836594 21706 consensus_queue.cc:237] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fb1ebcccf4248a49808a9e6b4b5ee08" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37815 } } peers { permanent_uuid: "f29bee8cc0374f7283a28d26b64309db" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40429 } } peers { permanent_uuid: "c527fa4543344ba1b99f1b84aaa76529" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36341 } }
I20260621 02:46:25.841174 21191 catalog_manager.cc:5495] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fb1ebcccf4248a49808a9e6b4b5ee08 (127.9.166.2). New cstate: current_term: 1 leader_uuid: "0fb1ebcccf4248a49808a9e6b4b5ee08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fb1ebcccf4248a49808a9e6b4b5ee08" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f29bee8cc0374f7283a28d26b64309db" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c527fa4543344ba1b99f1b84aaa76529" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36341 } health_report { overall_health: UNKNOWN } } }
I20260621 02:46:25.871172 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:25.865820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.9.166.2:37815)
0621 02:46:25.866132 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0621 02:46:25.866139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:46:25.866185 (+ 46us) server_negotiation.cc:407] Connection header received
0621 02:46:25.866440 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:46:25.866444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:46:25.866520 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:46:25.866670 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:46:25.867040 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.867758 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.869123 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.869501 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.869632 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:25.870118 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:46:25.870606 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:46:25.870769 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0621 02:46:25.870835 (+ 66us) server_negotiation.cc:299] Negotiation successful
0621 02:46:25.870912 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260621 02:46:25.878965 21636 raft_consensus.cc:1261] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db [term 1 FOLLOWER]: Refusing update from remote peer 0fb1ebcccf4248a49808a9e6b4b5ee08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:46:25.878963 21359 raft_consensus.cc:1261] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529 [term 1 FOLLOWER]: Refusing update from remote peer 0fb1ebcccf4248a49808a9e6b4b5ee08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:46:25.879943 21711 consensus_queue.cc:1035] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29bee8cc0374f7283a28d26b64309db" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:46:25.880237 21706 consensus_queue.cc:1035] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c527fa4543344ba1b99f1b84aaa76529" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:46:25.893153 21715 mvcc.cc:205] Tried to move back new op lower bound from 7299112902151217152 to 7299112901990449152. Current Snapshot: MvccSnapshot[applied={T|T < 7299112902151217152}]
I20260621 02:46:25.898486 21716 mvcc.cc:205] Tried to move back new op lower bound from 7299112902151217152 to 7299112901990449152. Current Snapshot: MvccSnapshot[applied={T|T < 7299112902151217152 or (T in {7299112902151217152})}]
I20260621 02:46:25.904961 21191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:46:25.905215 21191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:46:25.908775 21191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c90de5db10f04bbb85e547567555a6f5: Sending DeleteTablet for 3 replicas of tablet e4ee3e63fcd542be9b98c988af25317b
I20260621 02:46:25.909947 21616 tablet_service.cc:1514] Processing DeleteTablet for tablet e4ee3e63fcd542be9b98c988af25317b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:46:25 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:59908
I20260621 02:46:25.909996 21339 tablet_service.cc:1514] Processing DeleteTablet for tablet e4ee3e63fcd542be9b98c988af25317b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:46:25 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:44362
I20260621 02:46:25.910132 21482 tablet_service.cc:1514] Processing DeleteTablet for tablet e4ee3e63fcd542be9b98c988af25317b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:46:25 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:33332
I20260621 02:46:25.911823 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 21271
I20260621 02:46:25.919296 21725 tablet_replica.cc:330] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529: stopping tablet replica
I20260621 02:46:25.919783 21725 raft_consensus.cc:2229] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:46:25.920322 21725 raft_consensus.cc:2258] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:46:25.921125 21724 tablet_replica.cc:330] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db: stopping tablet replica
I20260621 02:46:25.921515 21724 raft_consensus.cc:2229] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:46:25.921986 21724 raft_consensus.cc:2258] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:46:25.922667 21726 tablet_replica.cc:330] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08: stopping tablet replica
I20260621 02:46:25.922928 21725 ts_tablet_manager.cc:1905] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:46:25.922992 21726 raft_consensus.cc:2229] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:46:25.923655 21726 raft_consensus.cc:2258] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:46:25.925455 21724 ts_tablet_manager.cc:1905] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:46:25.926018 21726 ts_tablet_manager.cc:1905] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:46:25.929849 21725 ts_tablet_manager.cc:1918] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Jun 21 02:46:25 dist-test-slave-hvgn krb5kdc[20715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:46:25.930011 21725 log.cc:1196] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/e4ee3e63fcd542be9b98c988af25317b
I20260621 02:46:25.930512 21725 ts_tablet_manager.cc:1939] T e4ee3e63fcd542be9b98c988af25317b P c527fa4543344ba1b99f1b84aaa76529: Deleting consensus metadata
I20260621 02:46:25.932729 21175 catalog_manager.cc:4841] TS c527fa4543344ba1b99f1b84aaa76529 (127.9.166.1:36341): tablet e4ee3e63fcd542be9b98c988af25317b (table test-table [id=43107d8092b146afb0a4e3f914f0434a]) successfully deleted
I20260621 02:46:25.932976 21726 ts_tablet_manager.cc:1918] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:46:25.933135 21726 log.cc:1196] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/e4ee3e63fcd542be9b98c988af25317b
I20260621 02:46:25.933591 21726 ts_tablet_manager.cc:1939] T e4ee3e63fcd542be9b98c988af25317b P 0fb1ebcccf4248a49808a9e6b4b5ee08: Deleting consensus metadata
I20260621 02:46:25.933701 21724 ts_tablet_manager.cc:1918] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:46:25.933859 21724 log.cc:1196] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/e4ee3e63fcd542be9b98c988af25317b
I20260621 02:46:25.934350 21724 ts_tablet_manager.cc:1939] T e4ee3e63fcd542be9b98c988af25317b P f29bee8cc0374f7283a28d26b64309db: Deleting consensus metadata
I20260621 02:46:25.935367 21178 catalog_manager.cc:4841] TS 0fb1ebcccf4248a49808a9e6b4b5ee08 (127.9.166.2:37815): tablet e4ee3e63fcd542be9b98c988af25317b (table test-table [id=43107d8092b146afb0a4e3f914f0434a]) successfully deleted
I20260621 02:46:25.936071 21178 catalog_manager.cc:4841] TS f29bee8cc0374f7283a28d26b64309db (127.9.166.3:40429): tablet e4ee3e63fcd542be9b98c988af25317b (table test-table [id=43107d8092b146afb0a4e3f914f0434a]) successfully deleted
I20260621 02:46:25.939153 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:25.913550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.9.166.1:36341)
0621 02:46:25.913779 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0621 02:46:25.913788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:46:25.918983 (+ 5195us) server_negotiation.cc:407] Connection header received
0621 02:46:25.919364 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:46:25.919370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:46:25.919548 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:46:25.919626 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:46:25.921870 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.922996 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.925113 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:25.925719 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:25.925813 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:25.931374 (+ 5561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:46:25.931417 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:46:25.931434 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:46:25.931478 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:46:25.936418 (+ 4940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:46:25.937121 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:46:25.937140 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:46:25.937143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:46:25.937267 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:46:25.937656 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:46:25.937661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:46:25.937665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:46:25.937993 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:46:25.938288 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0621 02:46:25.938596 (+ 308us) server_negotiation.cc:299] Negotiation successful
0621 02:46:25.938801 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:46:25.939805 21399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:46:26.033324 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 21271
I20260621 02:46:26.057433 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 21410
Jun 21 02:46:26 dist-test-slave-hvgn krb5kdc[20715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.2@KRBTEST.COM
I20260621 02:46:26.074543 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:26.058146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33346 (local address 127.9.166.2:37815)
0621 02:46:26.058314 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 02:46:26.058318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:46:26.058416 (+ 98us) server_negotiation.cc:407] Connection header received
0621 02:46:26.058606 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:46:26.058610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:46:26.058684 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:46:26.058766 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:46:26.060453 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:26.061291 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:26.063162 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:26.063703 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:26.063812 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:26.068761 (+ 4949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:46:26.068773 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:46:26.068788 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:46:26.068817 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:46:26.071768 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:46:26.072461 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:46:26.072473 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:46:26.072475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:46:26.072626 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:46:26.073020 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:46:26.073026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:46:26.073029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:46:26.073379 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:46:26.073670 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0621 02:46:26.073968 (+ 298us) server_negotiation.cc:299] Negotiation successful
0621 02:46:26.074171 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:46:26.075158 21542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:46:26.184105 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 21410
I20260621 02:46:26.201563 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 21553
Jun 21 02:46:26 dist-test-slave-hvgn krb5kdc[20715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.3@KRBTEST.COM
I20260621 02:46:26.216090 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:46:26.202325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59924 (local address 127.9.166.3:40429)
0621 02:46:26.202485 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0621 02:46:26.202489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:46:26.202507 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:46:26.202642 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:46:26.202646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:46:26.202717 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:46:26.202806 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:46:26.204140 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:26.204857 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:26.206077 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:46:26.206461 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:46:26.206574 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:46:26.210978 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:46:26.210992 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:46:26.211007 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:46:26.211040 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:46:26.213617 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:46:26.214267 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:46:26.214275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:46:26.214277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:46:26.214407 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:46:26.214715 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:46:26.214719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:46:26.214721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:46:26.215059 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:46:26.215269 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0621 02:46:26.215647 (+ 378us) server_negotiation.cc:299] Negotiation successful
0621 02:46:26.215847 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 02:46:26.216691 21676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:46:26.304486 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 21553
I20260621 02:46:26.321007 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 21159
I20260621 02:46:26.322130 21221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:46:26.432709 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 21159
I20260621 02:46:26.456017 9880 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260621 02:46:26.790658 9880 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260621 02:46:26.790941 9880 mini_ranger.cc:66] Stopping Ranger...
I20260621 02:46:27.135833 9880 mini_ranger.cc:68] Stopped Ranger
2026-06-21 02:46:27.136 UTC [20745] LOG: received smart shutdown request
2026-06-21 02:46:27.139 UTC [20745] LOG: background worker "logical replication launcher" (PID 20754) exited with exit code 1
2026-06-21 02:46:27.140 UTC [20749] LOG: shutting down
2026-06-21 02:46:27.285 UTC [20745] LOG: database system is shut down
2026-06-21T02:46:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47974 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:46:27 dist-test-slave-hvgn krb5kdc[21740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:46:27 dist-test-slave-hvgn krb5kdc[21740](info): set up 2 sockets
Jun 21 02:46:27 dist-test-slave-hvgn krb5kdc[21740](info): commencing operation
krb5kdc: starting...
W20260621 02:46:28.523938 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:46:28 dist-test-slave-hvgn krb5kdc[21740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782009988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:46:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:46:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.166.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.166.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.166.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/rangeradmin_127.9.166.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.166.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/rangeradmin_127.9.166.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.166.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.166.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.166.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/rangerlookup_127.9.166.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.166.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/rangerlookup_127.9.166.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.166.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.166.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.166.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/HTTP_127.9.166.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.166.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/HTTP_127.9.166.20@KRBTEST.COM.keytab.
I20260621 02:46:28.829989 9880 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task2e3lzq/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/postgres -l logfile start
2026-06-21 02:46:31.103 UTC [21770] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-21 02:46:31.103 UTC [21770] LOG: listening on IPv4 address "127.9.166.20", port 38161
2026-06-21 02:46:31.106 UTC [21770] LOG: listening on Unix socket "/tmp/.s.PGSQL.38161"
2026-06-21 02:46:31.127 UTC [21773] LOG: database system was shut down at 2026-06-21 02:46:29 UTC
2026-06-21 02:46:31.130 UTC [21770] LOG: database system is ready to accept connections
I20260621 02:46:32.115517 9880 mini_postgres.cc:96] Postgres bound to 38161
2026-06-21 02:46:32.137 UTC [21781] FATAL: database "slave" does not exist
127.9.166.20:38161 - accepting connections
I20260621 02:46:32.137723 9880 mini_ranger.cc:161] Starting Ranger...
I20260621 02:46:32.161720 9880 mini_ranger.cc:84] Created miniranger Postgres user
I20260621 02:46:32.565193 9880 mini_ranger.cc:87] Created ranger Postgres database
I20260621 02:46:32.565313 9880 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ranger-admin
2026-06-21 02:46:32,845 [I] DB FLAVOR :POSTGRES
2026-06-21 02:46:32,846 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 02:46:32,846 [I] Checking connection..
2026-06-21 02:46:32,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-21 02:46:33,103 [I] Checking connection passed.
2026-06-21 02:46:33,104 [I] --------- Verifying version history table ---------
2026-06-21 02:46:33,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 02:46:33,346 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 02:46:33,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 02:46:33,596 [I] Table x_db_version_h does not exist in database ranger
2026-06-21 02:46:33,597 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-21 02:46:33,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-21 02:46:33.842 UTC [21845] WARNING: there is no transaction in progress
2026-06-21 02:46:33,848 [I] create_dbversion_catalog.sql file imported successfully
2026-06-21 02:46:33,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-21 02:46:34,087 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-21 02:46:34,087 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 02:46:34,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-21 02:46:34,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-21 02:46:34,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hvgn.c.gcp-upstream.internal','N') ;"
2026-06-21 02:46:34,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 02:46:35,047 [I] Table x_portal_user does not exist in database ranger
2026-06-21 02:46:35,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 02:46:35,278 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-21 02:46:35,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 02:46:35,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 02:46:35,766 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-21 02:46:35,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-21 02:46:36.677 UTC [21953] WARNING: there is no transaction in progress
2026-06-21 02:46:36.692 UTC [21953] WARNING: there is no transaction in progress
2026-06-21 02:46:36.706 UTC [21953] WARNING: there is no transaction in progress
2026-06-21 02:46:36.720 UTC [21953] WARNING: there is no transaction in progress
2026-06-21 02:46:36.736 UTC [21953] WARNING: there is no transaction in progress
2026-06-21 02:46:36.756 UTC [21953] WARNING: there is no transaction in progress
2026-06-21 02:46:36.818 UTC [21953] WARNING: there is no transaction in progress
2026-06-21 02:46:36.824 UTC [21953] WARNING: there is no transaction in progress
2026-06-21 02:46:36.837 UTC [21953] WARNING: there is no transaction in progress
2026-06-21 02:46:36.847 UTC [21953] WARNING: there is no transaction in progress
2026-06-21 02:46:36.856 UTC [21953] WARNING: there is no transaction in progress
2026-06-21 02:46:36.869 UTC [21953] WARNING: there is no transaction in progress
2026-06-21 02:46:36.878 UTC [21953] WARNING: there is no transaction in progress
2026-06-21 02:46:36.884 UTC [21953] WARNING: there is no transaction in progress
2026-06-21 02:46:36.890 UTC [21953] WARNING: there is no transaction in progress
2026-06-21 02:46:36.896 UTC [21953] WARNING: there is no transaction in progress
2026-06-21 02:46:37,470 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-21 02:46:37,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-21 02:46:37,717 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-21 02:46:37,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-21 02:46:37,980 [I] Table x_portal_user already exists in database 'ranger'
2026-06-21 02:46:37,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-21 02:46:38,243 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-21 02:46:38,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-21 02:46:38,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-21 02:46:38,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hvgn.c.gcp-upstream.internal';"
2026-06-21 02:46:39,011 [I] CORE_DB_SCHEMA import status has been updated
2026-06-21 02:46:39,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-21 02:46:39,236 [I] DB_PATCHES have already been applied
I20260621 02:46:39.241968 9880 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ranger-admin:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task2e3lzq/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260621 02:46:39.242182 9880 mini_ranger.cc:193] Using host: 127.9.166.20
I20260621 02:46:39.256440 9880 mini_ranger.cc:227] Ranger admin URL: http://127.9.166.20:33335
Jun 21, 2026 2:46:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 2:46:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 2:46:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ranger-admin/ews/webapp
Jun 21, 2026 2:46:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 21, 2026 2:46:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 21, 2026 2:46:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ranger-admin/ews/webapp].
Jun 21, 2026 2:46:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260621 02:46:40.425833 9880 mini_ranger.cc:160] Time spent starting Ranger: real 8.288s user 0.000s sys 0.037s
Jun 21, 2026 2:47:00 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 21, 2026 2:47:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 21, 2026 2:47:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 21, 2026 2:47:00 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 21, 2026 2:47:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 21, 2026 2:47:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 21, 2026 2:47:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 21, 2026 2:47:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 21, 2026 2:47:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 21, 2026 2:47:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 21, 2026 2:47:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 21, 2026 2:47:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 21, 2026 2:47:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 21, 2026 2:47:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 21, 2026 2:47:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 21, 2026 2:47:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 21, 2026 2:47:00 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 21, 2026 2:47:01 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260621 02:47:02.145579 9880 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.166.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.166.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.166.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/rangerkms_127.9.166.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.166.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/rangerkms_127.9.166.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.166.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.166.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.166.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/HTTP_127.9.166.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.166.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/HTTP_127.9.166.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 21 02:47:02 dist-test-slave-hvgn krb5kdc[21740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010022, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260621 02:47:02.313381 9880 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260621 02:47:02.333514 9880 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260621 02:47:03.114490 9880 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260621 02:47:03.114615 9880 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ranger-kms
I20260621 02:47:03.114661 9880 mini_ranger_kms.cc:227] Using postgres at 127.9.166.20:38161
2026-06-21 02:47:03,463 [I] DB FLAVOR :POSTGRES
2026-06-21 02:47:03,464 [I] --------- Verifying Ranger DB connection ---------
2026-06-21 02:47:03,464 [I] Checking connection
2026-06-21 02:47:03,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-21 02:47:03,689 [I] connection success
2026-06-21 02:47:03,689 [I] --------- Verifying Ranger DB tables ---------
2026-06-21 02:47:03,689 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-21 02:47:03,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-21 02:47:03,927 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-21 02:47:03,927 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-21 02:47:03,928 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-21 02:47:03,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.166.20:38161/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-21 02:47:04,213 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260621 02:47:04.663250 9880 mini_ranger_kms.cc:310] Created kms service
I20260621 02:47:04.821556 9880 mini_ranger_kms.cc:326] Created kudu user
I20260621 02:47:04.865989 9880 mini_ranger_kms.cc:343] Created rangerkms user
Jun 21 02:47:04 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010022, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
I20260621 02:47:05.249814 9880 mini_ranger_kms.cc:384] Added ranger policy
I20260621 02:47:05.250131 9880 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ranger-kms:/tmp/dist-test-task2e3lzq/build/asan/bin/postgresql.jar:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task2e3lzq/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260621 02:47:05.250182 9880 mini_ranger_kms.cc:240] Using host: 127.9.166.20
I20260621 02:47:05.271556 9880 mini_ranger_kms.cc:276] Ranger KMS PID: 22133
I20260621 02:47:05.271680 9880 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.166.20:32969
Jun 21, 2026 2:47:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 21, 2026 2:47:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 21, 2026 2:47:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 21, 2026 2:47:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 21, 2026 2:47:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ranger-kms/ews/webapp].
Jun 21, 2026 2:47:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 21 02:47:07 dist-test-slave-hvgn krb5kdc[21740](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782010027, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.166.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:07.608084 9880 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.295s user 0.088s sys 0.278s
I20260621 02:47:07.608340 9880 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260621 02:47:07.608523 9880 mini_ranger_kms.cc:401] 127.9.166.20:32969/kms/v1/keys
Jun 21 02:47:07 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010027, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.166.20@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
Jun 21 02:47:08 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010027, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.166.20@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
Jun 21 02:47:10 dist-test-slave-hvgn krb5kdc[21740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:47:10.305436 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:36885
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:37703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:36885
--ranger_config_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.166.20:32969/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:47:10.631827 22184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:10.632175 22184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:10.632339 22184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:10.639783 22184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:47:10.639941 22184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:10.640055 22184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:10.640167 22184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:47:10.640247 22184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:47:10.650266 22184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:37703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.166.20:32969/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:36885
--ranger_config_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:36885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:10.651620 22184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:10.653059 22184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:10.665251 22190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:10.665295 22192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:10.665366 22189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:10.666004 22184 server_base.cc:1034] running on GCE node
I20260621 02:47:10.666786 22184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:10.668258 22184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:10.669497 22184 hybrid_clock.cc:648] HybridClock initialized: now 1782010030669462 us; error 43 us; skew 500 ppm
I20260621 02:47:10.669866 22184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:10 dist-test-slave-hvgn krb5kdc[21740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010030, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:10.672968 22184 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:47:10.674429 22184 webserver.cc:460] Webserver started at http://127.9.166.62:43319/ using document root <none> and password file <none>
I20260621 02:47:10.675148 22184 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:10.675277 22184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:10.675628 22184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:47:10 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010030, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
I20260621 02:47:10.724581 22184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "21d2ab7f3918407fb103e4c02d567ee5"
format_stamp: "Formatted at 2026-06-21 02:47:10 on dist-test-slave-hvgn"
server_key: "3fc27a477b66676fe84e40aaf8520bc7"
server_key_iv: "bb7f16d71c8aa8d4c057d65b0b27ac9d"
server_key_version: "kuduclusterkey@0"
I20260621 02:47:10.725476 22184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "21d2ab7f3918407fb103e4c02d567ee5"
format_stamp: "Formatted at 2026-06-21 02:47:10 on dist-test-slave-hvgn"
server_key: "3fc27a477b66676fe84e40aaf8520bc7"
server_key_iv: "bb7f16d71c8aa8d4c057d65b0b27ac9d"
server_key_version: "kuduclusterkey@0"
I20260621 02:47:10.731104 22184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:47:10.747067 22199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:10.748710 22184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:47:10.748914 22184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "21d2ab7f3918407fb103e4c02d567ee5"
format_stamp: "Formatted at 2026-06-21 02:47:10 on dist-test-slave-hvgn"
server_key: "3fc27a477b66676fe84e40aaf8520bc7"
server_key_iv: "bb7f16d71c8aa8d4c057d65b0b27ac9d"
server_key_version: "kuduclusterkey@0"
I20260621 02:47:10.749114 22184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:10.763513 22184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:10.767263 22184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:10.767602 22184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:10.786844 22184 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:36885
I20260621 02:47:10.786865 22250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:36885 every 8 connection(s)
I20260621 02:47:10.788244 22184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:47:10.789791 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 22184
I20260621 02:47:10.790016 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:47:10.790666 9880 external_mini_cluster.cc:1428] Setting key 15e8506d514c4d45c2646a80d27821ed
I20260621 02:47:10.792768 22251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:47:10.806638 22251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5: Bootstrap starting.
I20260621 02:47:10.810191 22251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5: Neither blocks nor log segments found. Creating new log.
I20260621 02:47:10.811169 22251 log.cc:824] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5: Log is configured to *not* fsync() on all Append() calls
Jun 21 02:47:10 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:10.814491 22251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5: No bootstrap required, opened a new log
I20260621 02:47:10.821312 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:10.792824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36026 (local address 127.9.166.62:36885)
0621 02:47:10.799932 (+ 7108us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:10.799949 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:10.799998 (+ 49us) server_negotiation.cc:407] Connection header received
0621 02:47:10.802308 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:10.802336 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:10.802842 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:10.803277 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:10.804477 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:10.806347 (+ 1870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:10.807734 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:10.808826 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:10.809009 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:10.813053 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:10.813064 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:10.813082 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:10.813126 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:10.816994 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:10.817631 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:10.817640 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:10.817650 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:10.817870 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:10.818213 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:10.818218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:10.818221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:10.818914 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:10.819235 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:10.819563 (+ 328us) server_negotiation.cc:299] Negotiation successful
0621 02:47:10.819974 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6858,"thread_start_us":6607,"threads_started":1}
I20260621 02:47:10.824807 22251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d2ab7f3918407fb103e4c02d567ee5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36885 } }
I20260621 02:47:10.825117 22251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:47:10.825191 22251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d2ab7f3918407fb103e4c02d567ee5, State: Initialized, Role: FOLLOWER
I20260621 02:47:10.825651 22251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d2ab7f3918407fb103e4c02d567ee5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36885 } }
I20260621 02:47:10.825842 22251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:47:10.825937 22251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:47:10.826045 22251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:47:10.828102 22251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d2ab7f3918407fb103e4c02d567ee5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36885 } }
I20260621 02:47:10.828524 22251 leader_election.cc:302] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21d2ab7f3918407fb103e4c02d567ee5; no voters:
I20260621 02:47:10.829169 22251 leader_election.cc:288] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:47:10.829296 22256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:47:10.829967 22256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [term 1 LEADER]: Becoming Leader. State: Replica: 21d2ab7f3918407fb103e4c02d567ee5, State: Running, Role: LEADER
I20260621 02:47:10.830488 22256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d2ab7f3918407fb103e4c02d567ee5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36885 } }
I20260621 02:47:10.831142 22251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:47:10.833168 22257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21d2ab7f3918407fb103e4c02d567ee5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d2ab7f3918407fb103e4c02d567ee5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36885 } } }
I20260621 02:47:10.833196 22258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21d2ab7f3918407fb103e4c02d567ee5. Latest consensus state: current_term: 1 leader_uuid: "21d2ab7f3918407fb103e4c02d567ee5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d2ab7f3918407fb103e4c02d567ee5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36885 } } }
I20260621 02:47:10.833622 22257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [sys.catalog]: This master's current role is: LEADER
I20260621 02:47:10.833628 22258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [sys.catalog]: This master's current role is: LEADER
I20260621 02:47:10.834978 22261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:47:10.840524 22261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:47:10.841959 22251 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 02:47:10.857017 22261 catalog_manager.cc:1347] Generated new cluster ID: 9786af470f2b465cb01991407975048e
I20260621 02:47:10.857203 22261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:47:10.877888 22261 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:47:10.879367 22261 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:47:10.895030 22261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5: Generated new TSK 0
I20260621 02:47:10.896327 22261 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
I20260621 02:47:11.602140 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.166.20:32969/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:36885
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:37703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:47:11 dist-test-slave-hvgn krb5kdc[21740](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782010031, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260621 02:47:11.997896 22296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:11.998273 22296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:11.998481 22296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:12.010002 22296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:12.010146 22296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:12.010389 22296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:47:12.022454 22296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:37703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.166.20:32969/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:12.023948 22296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:12.025363 22296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:12.042451 22307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:12.043752 22306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:12.045065 22309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:12.045444 22296 server_base.cc:1034] running on GCE node
I20260621 02:47:12.046147 22296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:12.047333 22296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:12.048676 22296 hybrid_clock.cc:648] HybridClock initialized: now 1782010032048607 us; error 56 us; skew 500 ppm
I20260621 02:47:12.049108 22296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:12 dist-test-slave-hvgn krb5kdc[21740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010032, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:12.053197 22296 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:47:12.055011 22296 webserver.cc:460] Webserver started at http://127.9.166.1:39763/ using document root <none> and password file <none>
I20260621 02:47:12.055934 22296 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:12.056077 22296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:12.056396 22296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:47:12 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010032, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
I20260621 02:47:12.075407 22296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "cdab9cda14cf4edd9b36302e1c140181"
format_stamp: "Formatted at 2026-06-21 02:47:12 on dist-test-slave-hvgn"
server_key: "4dee75c0a424f0f20c37efc8504bc682"
server_key_iv: "8f697cce760e6d1d46c78580bd9fa28a"
server_key_version: "kuduclusterkey@0"
I20260621 02:47:12.078249 22296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "cdab9cda14cf4edd9b36302e1c140181"
format_stamp: "Formatted at 2026-06-21 02:47:12 on dist-test-slave-hvgn"
server_key: "4dee75c0a424f0f20c37efc8504bc682"
server_key_iv: "8f697cce760e6d1d46c78580bd9fa28a"
server_key_version: "kuduclusterkey@0"
I20260621 02:47:12.085625 22296 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260621 02:47:12.112421 22316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:12.117611 22296 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.008s sys 0.002s
I20260621 02:47:12.117919 22296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "cdab9cda14cf4edd9b36302e1c140181"
format_stamp: "Formatted at 2026-06-21 02:47:12 on dist-test-slave-hvgn"
server_key: "4dee75c0a424f0f20c37efc8504bc682"
server_key_iv: "8f697cce760e6d1d46c78580bd9fa28a"
server_key_version: "kuduclusterkey@0"
I20260621 02:47:12.118299 22296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:12.149354 22296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:12.155064 22296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:12.155508 22296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:12.156792 22296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:47:12.159308 22296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:47:12.159487 22296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260621 02:47:12.159686 22296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:47:12.159813 22296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:12.219805 22296 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:38297
I20260621 02:47:12.219820 22428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:38297 every 8 connection(s)
I20260621 02:47:12.221735 22296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:47:12.224951 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 22296
I20260621 02:47:12.225225 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:47:12.226243 9880 external_mini_cluster.cc:1428] Setting key 67c45fea8e0edad8261dc5e27a61eca8
Jun 21 02:47:12 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010032, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.2@KRBTEST.COM" created.
I20260621 02:47:12.269181 22432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:12.226198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:41145 (local address 127.9.166.62:36885)
0621 02:47:12.239575 (+ 13377us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:12.239584 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:12.239604 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:47:12.239677 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:12.239682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:12.239782 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:12.239896 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:12.247566 (+ 7670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:12.248790 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:12.251466 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:12.252129 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:12.252260 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:12.259404 (+ 7144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:12.259412 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:12.259417 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:12.259461 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:12.263410 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:12.264270 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:12.264276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:12.264280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:12.264466 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:12.265037 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:12.265043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:12.265046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:12.265478 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:12.265631 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:12.268397 (+ 2766us) server_negotiation.cc:299] Negotiation successful
0621 02:47:12.268650 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13229,"thread_start_us":14962,"threads_started":1}
I20260621 02:47:12.271203 22431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:12.225386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:36885 (local address 127.9.166.1:41145)
0621 02:47:12.236061 (+ 10675us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:12.236113 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:12.238844 (+ 2731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:12.241214 (+ 2370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:12.241227 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:12.242008 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:12.242830 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:12.242855 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:12.248995 (+ 6140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:12.249002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:12.251273 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:12.251290 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:12.254291 (+ 3001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:12.254312 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:12.254627 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:12.255897 (+ 1270us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:12.255935 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:12.259107 (+ 3172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:12.263572 (+ 4465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:12.263582 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:12.263600 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:12.264132 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:12.264617 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:12.264621 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:12.264626 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:12.264895 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:12.265703 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:12.265715 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:12.266193 (+ 478us) client_negotiation.cc:769] Sending connection context
0621 02:47:12.266499 (+ 306us) client_negotiation.cc:240] Negotiation successful
0621 02:47:12.269590 (+ 3091us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10274,"spinlock_wait_cycles":24320,"thread_start_us":10168,"threads_started":1}
I20260621 02:47:12.273466 22429 heartbeater.cc:347] Connected to a master server at 127.9.166.62:36885
I20260621 02:47:12.273788 22429 heartbeater.cc:464] Registering TS with master...
I20260621 02:47:12.274670 22429 heartbeater.cc:511] Master 127.9.166.62:36885 requested a full tablet report, sending...
I20260621 02:47:12.277276 22216 ts_manager.cc:194] Registered new tserver with Master: cdab9cda14cf4edd9b36302e1c140181 (127.9.166.1:38297)
I20260621 02:47:12.279754 22216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:41145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
I20260621 02:47:12.297034 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.166.20:32969/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:36885
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:37703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:47:12 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010031, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
Jun 21 02:47:12 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010031, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
W20260621 02:47:12.747138 22435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:12.747650 22435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:12.747849 22435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:12.759368 22435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:12.759601 22435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:12.759820 22435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:47:12.775419 22435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:37703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.166.20:32969/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:12.777243 22435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:12.778930 22435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:12.828951 22443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:12.829582 22442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:12.829746 22445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:12.831283 22435 server_base.cc:1034] running on GCE node
I20260621 02:47:12.832077 22435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:12.833313 22435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:12.834528 22435 hybrid_clock.cc:648] HybridClock initialized: now 1782010032834468 us; error 57 us; skew 500 ppm
I20260621 02:47:12.834867 22435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:12 dist-test-slave-hvgn krb5kdc[21740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010032, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:12.838661 22435 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:47:12.841018 22435 webserver.cc:460] Webserver started at http://127.9.166.2:38881/ using document root <none> and password file <none>
I20260621 02:47:12.841969 22435 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:12.842114 22435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:12.842500 22435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:47:12 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010032, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
I20260621 02:47:12.867147 22435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "841ce961c8b843f3a40579c70c1a0b48"
format_stamp: "Formatted at 2026-06-21 02:47:12 on dist-test-slave-hvgn"
server_key: "23ee53c02e0ec4bfe9cd00a0af1022b4"
server_key_iv: "b0845b6fd4340c5b5f63ce217ffaae50"
server_key_version: "kuduclusterkey@0"
I20260621 02:47:12.877265 22435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "841ce961c8b843f3a40579c70c1a0b48"
format_stamp: "Formatted at 2026-06-21 02:47:12 on dist-test-slave-hvgn"
server_key: "23ee53c02e0ec4bfe9cd00a0af1022b4"
server_key_iv: "b0845b6fd4340c5b5f63ce217ffaae50"
server_key_version: "kuduclusterkey@0"
I20260621 02:47:12.889533 22435 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.006s sys 0.003s
I20260621 02:47:12.919735 22452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:12.921756 22435 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260621 02:47:12.922021 22435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "841ce961c8b843f3a40579c70c1a0b48"
format_stamp: "Formatted at 2026-06-21 02:47:12 on dist-test-slave-hvgn"
server_key: "23ee53c02e0ec4bfe9cd00a0af1022b4"
server_key_iv: "b0845b6fd4340c5b5f63ce217ffaae50"
server_key_version: "kuduclusterkey@0"
I20260621 02:47:12.922369 22435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:12.940729 22435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:12.944782 22435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:12.945093 22435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:12.946045 22435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:47:12.947507 22435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:47:12.947579 22435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:12.947742 22435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:47:12.947790 22435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:12.981838 22435 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:44573
I20260621 02:47:12.981926 22571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:44573 every 8 connection(s)
I20260621 02:47:12.983121 22435 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:47:12.991827 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 22435
I20260621 02:47:12.992025 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:47:12.992550 9880 external_mini_cluster.cc:1428] Setting key 09c479ea0424ee95c3e72a8a853a089e
Jun 21 02:47:12 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010032, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:13.007342 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:12.986675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:36885 (local address 127.9.166.2:34977)
0621 02:47:12.987516 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:12.987567 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:12.989230 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:12.989931 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:12.989939 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:12.990345 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:12.990916 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:12.990932 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:12.992447 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:12.992452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:12.994080 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:12.994091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:12.995075 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:12.995082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:12.995369 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:12.996228 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:12.996258 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:12.999158 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:13.003172 (+ 4014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:13.003179 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:13.003194 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:13.003799 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:13.004201 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:13.004206 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:13.004210 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:13.004420 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:13.004971 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:13.004980 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:13.005434 (+ 454us) client_negotiation.cc:769] Sending connection context
0621 02:47:13.005787 (+ 353us) client_negotiation.cc:240] Negotiation successful
0621 02:47:13.006269 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":374,"threads_started":1}
I20260621 02:47:13.008265 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:12.986064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:34977 (local address 127.9.166.62:36885)
0621 02:47:12.986580 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:12.986588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:12.987704 (+ 1116us) server_negotiation.cc:407] Connection header received
0621 02:47:12.989496 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:12.989502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:12.989586 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:12.989715 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:12.991093 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:12.992278 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:12.994231 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:12.994840 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:12.994970 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:12.999444 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:12.999453 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:12.999458 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:12.999498 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:13.003003 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:13.003935 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:13.003941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:13.003946 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:13.004076 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:13.004549 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:13.004554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:13.004565 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:13.004843 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:13.005005 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:13.007637 (+ 2632us) server_negotiation.cc:299] Negotiation successful
0621 02:47:13.007874 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":258,"threads_started":1}
I20260621 02:47:13.009526 22572 heartbeater.cc:347] Connected to a master server at 127.9.166.62:36885
I20260621 02:47:13.009850 22572 heartbeater.cc:464] Registering TS with master...
I20260621 02:47:13.010592 22572 heartbeater.cc:511] Master 127.9.166.62:36885 requested a full tablet report, sending...
I20260621 02:47:13.012115 22216 ts_manager.cc:194] Registered new tserver with Master: 841ce961c8b843f3a40579c70c1a0b48 (127.9.166.2:44573)
I20260621 02:47:13.012944 22216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:34977
WARNING: no policy specified for kudu/127.9.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
I20260621 02:47:13.051622 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.166.20:32969/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:36885
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:37703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:47:13.283718 22429 heartbeater.cc:503] Master 127.9.166.62:36885 was elected leader, sending a full tablet report...
W20260621 02:47:13.395602 22578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:13.395862 22578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:13.395951 22578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:13.402940 22578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:13.403031 22578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:13.403133 22578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:47:13.412380 22578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:37703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.166.20:32969/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:13.413539 22578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:13.414791 22578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:13.427525 22583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:13.427474 22586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:13.427583 22584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:13.428757 22578 server_base.cc:1034] running on GCE node
I20260621 02:47:13.429315 22578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:13.430130 22578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:13.431413 22578 hybrid_clock.cc:648] HybridClock initialized: now 1782010033431345 us; error 55 us; skew 500 ppm
I20260621 02:47:13.431747 22578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:13 dist-test-slave-hvgn krb5kdc[21740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010033, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:13.439962 22578 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:47:13.441440 22578 webserver.cc:460] Webserver started at http://127.9.166.3:42399/ using document root <none> and password file <none>
I20260621 02:47:13.442142 22578 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:13.442283 22578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:13.442572 22578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 21 02:47:13 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010033, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
I20260621 02:47:13.458518 22578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "0896084bbfb34282b5071fc849b9b00c"
format_stamp: "Formatted at 2026-06-21 02:47:13 on dist-test-slave-hvgn"
server_key: "f99def0660dd89d276145ee507bc1c53"
server_key_iv: "fd99fb32a7abb1c11f8d8ed35da214c8"
server_key_version: "kuduclusterkey@0"
I20260621 02:47:13.459489 22578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "0896084bbfb34282b5071fc849b9b00c"
format_stamp: "Formatted at 2026-06-21 02:47:13 on dist-test-slave-hvgn"
server_key: "f99def0660dd89d276145ee507bc1c53"
server_key_iv: "fd99fb32a7abb1c11f8d8ed35da214c8"
server_key_version: "kuduclusterkey@0"
I20260621 02:47:13.466965 22578 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20260621 02:47:13.481447 22593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:13.483068 22578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:47:13.483263 22578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "0896084bbfb34282b5071fc849b9b00c"
format_stamp: "Formatted at 2026-06-21 02:47:13 on dist-test-slave-hvgn"
server_key: "f99def0660dd89d276145ee507bc1c53"
server_key_iv: "fd99fb32a7abb1c11f8d8ed35da214c8"
server_key_version: "kuduclusterkey@0"
I20260621 02:47:13.483552 22578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:13.495509 22578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:13.498874 22578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:13.499176 22578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:13.500084 22578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:47:13.501346 22578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:47:13.501448 22578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:13.501595 22578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:47:13.501668 22578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:13.534808 22578 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:43219
I20260621 02:47:13.534899 22705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:43219 every 8 connection(s)
I20260621 02:47:13.536227 22578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:47:13.536741 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 22578
I20260621 02:47:13.536959 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:47:13.537611 9880 external_mini_cluster.cc:1428] Setting key d3b7c52c4af7a3f85c3e74cf2d963679
Jun 21 02:47:13 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010033, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:13.556634 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:13.539044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:60301 (local address 127.9.166.62:36885)
0621 02:47:13.539504 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:13.539510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:13.540769 (+ 1259us) server_negotiation.cc:407] Connection header received
0621 02:47:13.542587 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:13.542592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:13.542667 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:13.542750 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:13.544283 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:13.545060 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:13.546527 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:13.546937 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:13.547032 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:13.550257 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:13.550262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:13.550266 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:13.550305 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:13.553665 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:13.554354 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:13.554359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:13.554363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:13.554498 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:13.554928 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:13.554933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:13.554936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:13.555218 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:13.555374 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:13.556069 (+ 695us) server_negotiation.cc:299] Negotiation successful
0621 02:47:13.556286 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":239,"threads_started":1}
I20260621 02:47:13.557564 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:13.539743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:36885 (local address 127.9.166.3:60301)
0621 02:47:13.540616 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:13.540662 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:13.542299 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:13.542950 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:13.542957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:13.543357 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:13.544129 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:13.544148 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:13.545195 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:13.545199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:13.546387 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:13.546398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:13.547124 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:13.547128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:13.547317 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:13.548071 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:13.548092 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:13.550042 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:13.553797 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:13.553805 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:13.553828 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:13.554228 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:13.554607 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:13.554612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:13.554616 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:13.554807 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:13.555346 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:13.555353 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:13.555715 (+ 362us) client_negotiation.cc:769] Sending connection context
0621 02:47:13.555968 (+ 253us) client_negotiation.cc:240] Negotiation successful
0621 02:47:13.556447 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":389,"threads_started":1}
I20260621 02:47:13.558862 22706 heartbeater.cc:347] Connected to a master server at 127.9.166.62:36885
I20260621 02:47:13.559167 22706 heartbeater.cc:464] Registering TS with master...
I20260621 02:47:13.559927 22706 heartbeater.cc:511] Master 127.9.166.62:36885 requested a full tablet report, sending...
I20260621 02:47:13.561287 22216 ts_manager.cc:194] Registered new tserver with Master: 0896084bbfb34282b5071fc849b9b00c (127.9.166.3:43219)
I20260621 02:47:13.562029 22216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:60301
I20260621 02:47:13.565693 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:47:13.567334 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 22296
Jun 21 02:47:13 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:47:13.584951 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:13.568313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.9.166.1:38297)
0621 02:47:13.568982 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:13.568991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:13.569022 (+ 31us) server_negotiation.cc:407] Connection header received
0621 02:47:13.569079 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:13.569085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:13.569268 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:13.569435 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:13.570840 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:13.571932 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:13.573331 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:13.573865 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:13.574011 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:13.578574 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:13.578584 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:13.578597 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:13.578628 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:13.581848 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:13.582560 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:13.582567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:13.582569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:13.582717 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:13.583032 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:13.583035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:13.583036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:13.583396 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:13.583802 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:13.584173 (+ 371us) server_negotiation.cc:299] Negotiation successful
0621 02:47:13.584450 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":339,"threads_started":1}
I20260621 02:47:13.586197 22424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:13.669157 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 22296
I20260621 02:47:13.684854 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 22435
Jun 21 02:47:13 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.2@KRBTEST.COM
I20260621 02:47:13.700722 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:13.685825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.9.166.2:44573)
0621 02:47:13.686345 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:13.686358 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:13.686383 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:47:13.686442 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:13.686448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:13.686654 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:13.686788 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:13.688188 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:13.689213 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:13.690495 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:13.691023 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:13.691122 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:13.695347 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:13.695357 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:13.695372 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:13.695466 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:13.698095 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:13.698720 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:13.698727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:13.698730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:13.698867 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:13.699183 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:13.699186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:13.699188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:13.699718 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:13.699944 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:13.700237 (+ 293us) server_negotiation.cc:299] Negotiation successful
0621 02:47:13.700433 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":286,"threads_started":1}
I20260621 02:47:13.702019 22567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:13.784283 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 22435
I20260621 02:47:13.804987 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 22578
Jun 21 02:47:13 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.3@KRBTEST.COM
I20260621 02:47:13.821139 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:13.805948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51616 (local address 127.9.166.3:43219)
0621 02:47:13.806621 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:13.806627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:13.806655 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:47:13.806711 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:13.806717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:13.806917 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:13.807063 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:13.808496 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:13.809464 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:13.810661 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:13.811212 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:13.811307 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:13.815616 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:13.815626 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:13.815640 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:13.815675 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:13.818467 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:13.819055 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:13.819062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:13.819064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:13.819191 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:13.819646 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:13.819651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:13.819654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:13.820023 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:13.820279 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:13.820514 (+ 235us) server_negotiation.cc:299] Negotiation successful
0621 02:47:13.820765 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":362,"threads_started":1}
I20260621 02:47:13.822535 22701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:13.905722 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 22578
I20260621 02:47:13.921659 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 22184
I20260621 02:47:13.923022 22246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:14.032998 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 22184
I20260621 02:47:14.056643 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:36885
--webserver_interface=127.9.166.62
--webserver_port=43319
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:37703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:36885
--ranger_config_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.166.20:32969/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.166.20:32969/kms with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:47:14.376048 22718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:14.376332 22718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:14.376418 22718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:14.383528 22718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:47:14.383630 22718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:14.383673 22718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:14.383728 22718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:47:14.383767 22718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:47:14.393604 22718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:37703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.166.20:32969/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:36885
--ranger_config_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:36885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=43319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:14.394790 22718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:14.396044 22718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:14.407742 22723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:14.407663 22724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:14.407711 22726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:14.408833 22718 server_base.cc:1034] running on GCE node
I20260621 02:47:14.409651 22718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:14.411031 22718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:14.412283 22718 hybrid_clock.cc:648] HybridClock initialized: now 1782010034412220 us; error 53 us; skew 500 ppm
I20260621 02:47:14.412648 22718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:14 dist-test-slave-hvgn krb5kdc[21740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010034, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:14.415711 22718 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:47:14.417133 22718 webserver.cc:460] Webserver started at http://127.9.166.62:43319/ using document root <none> and password file <none>
I20260621 02:47:14.417774 22718 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:14.417925 22718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 02:47:14 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010034, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
I20260621 02:47:14.471522 22718 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:47:14.474895 22734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:14.476522 22718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:47:14.476742 22718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "21d2ab7f3918407fb103e4c02d567ee5"
format_stamp: "Formatted at 2026-06-21 02:47:10 on dist-test-slave-hvgn"
server_key: "3fc27a477b66676fe84e40aaf8520bc7"
server_key_iv: "bb7f16d71c8aa8d4c057d65b0b27ac9d"
server_key_version: "kuduclusterkey@0"
I20260621 02:47:14.477818 22718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:14.524900 22718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:14.528928 22718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:14.529218 22718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:14.547442 22718 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:36885
I20260621 02:47:14.547467 22785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:36885 every 8 connection(s)
I20260621 02:47:14.549423 22718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:47:14.555962 22786 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:47:14.558976 22786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5: Bootstrap starting.
I20260621 02:47:14.559031 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 22718
I20260621 02:47:14.560026 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.166.20:32969/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:38297
--local_ip_for_outbound_sockets=127.9.166.1
--tserver_master_addrs=127.9.166.62:36885
--webserver_port=39763
--webserver_interface=127.9.166.1
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:37703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.166.20:32969/kms with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:47:14.582346 22786 log.cc:824] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:14.593839 22786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:47:14.594501 22786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5: Bootstrap complete.
I20260621 02:47:14.607489 22786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d2ab7f3918407fb103e4c02d567ee5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36885 } }
I20260621 02:47:14.608661 22786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d2ab7f3918407fb103e4c02d567ee5, State: Initialized, Role: FOLLOWER
I20260621 02:47:14.609380 22786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d2ab7f3918407fb103e4c02d567ee5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36885 } }
I20260621 02:47:14.609654 22786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:47:14.609767 22786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:47:14.609943 22786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [term 1 FOLLOWER]: Advancing to term 2
I20260621 02:47:14.615209 22786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d2ab7f3918407fb103e4c02d567ee5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36885 } }
I20260621 02:47:14.615840 22786 leader_election.cc:302] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21d2ab7f3918407fb103e4c02d567ee5; no voters:
I20260621 02:47:14.616498 22786 leader_election.cc:288] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 02:47:14.616626 22790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 02:47:14.617926 22790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [term 2 LEADER]: Becoming Leader. State: Replica: 21d2ab7f3918407fb103e4c02d567ee5, State: Running, Role: LEADER
I20260621 02:47:14.618441 22790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d2ab7f3918407fb103e4c02d567ee5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36885 } }
I20260621 02:47:14.618651 22786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:47:14.621151 22792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21d2ab7f3918407fb103e4c02d567ee5. Latest consensus state: current_term: 2 leader_uuid: "21d2ab7f3918407fb103e4c02d567ee5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d2ab7f3918407fb103e4c02d567ee5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36885 } } }
I20260621 02:47:14.621577 22792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [sys.catalog]: This master's current role is: LEADER
I20260621 02:47:14.621173 22791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "21d2ab7f3918407fb103e4c02d567ee5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d2ab7f3918407fb103e4c02d567ee5" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36885 } } }
I20260621 02:47:14.621724 22791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5 [sys.catalog]: This master's current role is: LEADER
I20260621 02:47:14.627542 22798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:47:14.638130 22798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:47:14.638885 22786 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260621 02:47:14.639562 22798 catalog_manager.cc:1259] Loaded cluster ID: 9786af470f2b465cb01991407975048e
I20260621 02:47:14.639681 22798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:47:14.643553 22798 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:47:14.644927 22798 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5: Loaded TSK: 0
I20260621 02:47:14.646517 22798 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 02:47:15.046136 22788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:15.046635 22788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:15.046838 22788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:15.057724 22788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:15.057941 22788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:15.058216 22788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:47:15.072636 22788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:37703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.166.20:32969/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:38297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=39763
--tserver_master_addrs=127.9.166.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:15.074139 22788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:15.075735 22788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:15.099151 22824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:15.100783 22823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:15.105209 22788 server_base.cc:1034] running on GCE node
W20260621 02:47:15.105407 22826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:15.106231 22788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:15.107474 22788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:15.108742 22788 hybrid_clock.cc:648] HybridClock initialized: now 1782010035108693 us; error 34 us; skew 500 ppm
I20260621 02:47:15.109143 22788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:15 dist-test-slave-hvgn krb5kdc[21740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010035, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:15.113396 22788 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:47:15.115363 22788 webserver.cc:460] Webserver started at http://127.9.166.1:39763/ using document root <none> and password file <none>
I20260621 02:47:15.116353 22788 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:15.116534 22788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 02:47:15 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010035, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
I20260621 02:47:15.148187 22788 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:47:15.156023 22834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:15.157898 22788 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.006s sys 0.000s
I20260621 02:47:15.158188 22788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "cdab9cda14cf4edd9b36302e1c140181"
format_stamp: "Formatted at 2026-06-21 02:47:12 on dist-test-slave-hvgn"
server_key: "4dee75c0a424f0f20c37efc8504bc682"
server_key_iv: "8f697cce760e6d1d46c78580bd9fa28a"
server_key_version: "kuduclusterkey@0"
I20260621 02:47:15.159744 22788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:15.181941 22788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:15.186363 22788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:15.186666 22788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:15.187695 22788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:47:15.189122 22788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:47:15.189239 22788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:15.189371 22788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:47:15.189469 22788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:15.256369 22788 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:38297
I20260621 02:47:15.256448 22947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:38297 every 8 connection(s)
I20260621 02:47:15.258369 22788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:47:15.264830 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 22788
I20260621 02:47:15.265620 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.166.20:32969/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:44573
--local_ip_for_outbound_sockets=127.9.166.2
--tserver_master_addrs=127.9.166.62:36885
--webserver_port=38881
--webserver_interface=127.9.166.2
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:37703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.166.20:32969/kms with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:47:15 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010035, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:15.317554 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:15.261845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:36885 (local address 127.9.166.1:40205)
0621 02:47:15.262681 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:15.262723 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:15.275254 (+ 12531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:15.281826 (+ 6572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:15.281853 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:15.282519 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:15.283503 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:15.283534 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:15.286663 (+ 3129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:15.286672 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:15.289179 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:15.289200 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:15.292180 (+ 2980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:15.292188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:15.292480 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:15.293405 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:15.293442 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:15.296539 (+ 3097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:15.312257 (+ 15718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:15.312267 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:15.312289 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:15.312956 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:15.313569 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:15.313575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:15.313579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:15.313872 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:15.314892 (+ 1020us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:15.314901 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:15.315290 (+ 389us) client_negotiation.cc:769] Sending connection context
0621 02:47:15.315645 (+ 355us) client_negotiation.cc:240] Negotiation successful
0621 02:47:15.316222 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":369,"threads_started":1}
I20260621 02:47:15.321801 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:15.261603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:40205 (local address 127.9.166.62:36885)
0621 02:47:15.262316 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:15.262338 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:15.262908 (+ 570us) server_negotiation.cc:407] Connection header received
0621 02:47:15.280018 (+ 17110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:15.280045 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:15.280665 (+ 620us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:15.283793 (+ 3128us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:15.284300 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:15.286461 (+ 2161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:15.290602 (+ 4141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:15.291885 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:15.293014 (+ 1129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:15.307590 (+ 14576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:15.307602 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:15.307624 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:15.307673 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:15.312031 (+ 4358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:15.313192 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:15.313201 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:15.313209 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:15.313400 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:15.314013 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:15.314019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:15.314022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:15.314688 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:15.319066 (+ 4378us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:15.319512 (+ 446us) server_negotiation.cc:299] Negotiation successful
0621 02:47:15.320371 (+ 859us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":276,"threads_started":1}
I20260621 02:47:15.325665 22948 heartbeater.cc:347] Connected to a master server at 127.9.166.62:36885
I20260621 02:47:15.325970 22948 heartbeater.cc:464] Registering TS with master...
I20260621 02:47:15.326654 22948 heartbeater.cc:511] Master 127.9.166.62:36885 requested a full tablet report, sending...
W20260621 02:47:15.329874 22948 heartbeater.cc:644] Failed to heartbeat to 127.9.166.62:36885 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 02:47:15.330116 22948 heartbeater.cc:464] Registering TS with master...
I20260621 02:47:15.330387 22948 heartbeater.cc:511] Master 127.9.166.62:36885 requested a full tablet report, sending...
I20260621 02:47:15.331704 22948 heartbeater.cc:464] Registering TS with master...
I20260621 02:47:15.332021 22948 heartbeater.cc:511] Master 127.9.166.62:36885 requested a full tablet report, sending...
W20260621 02:47:15.333271 22948 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 02:47:15.673877 22952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:15.674134 22952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:15.674281 22952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:15.681572 22952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:15.681738 22952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:15.681928 22952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:47:15.694926 22952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:37703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.166.20:32969/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:44573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=38881
--tserver_master_addrs=127.9.166.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:15.696267 22952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:15.697659 22952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:15.713650 22969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:15.716564 22967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:15.716611 22966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:15.718078 22952 server_base.cc:1034] running on GCE node
I20260621 02:47:15.718793 22952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:15.719978 22952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:15.721308 22952 hybrid_clock.cc:648] HybridClock initialized: now 1782010035721232 us; error 56 us; skew 500 ppm
I20260621 02:47:15.721643 22952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:15 dist-test-slave-hvgn krb5kdc[21740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010035, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:15.724853 22952 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:47:15.730989 22952 webserver.cc:460] Webserver started at http://127.9.166.2:38881/ using document root <none> and password file <none>
I20260621 02:47:15.731788 22952 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:15.731946 22952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 02:47:15 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010035, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
I20260621 02:47:15.764134 22952 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.005s sys 0.008s
I20260621 02:47:15.779425 22977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:15.781522 22952 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.007s sys 0.008s
I20260621 02:47:15.781720 22952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "841ce961c8b843f3a40579c70c1a0b48"
format_stamp: "Formatted at 2026-06-21 02:47:12 on dist-test-slave-hvgn"
server_key: "23ee53c02e0ec4bfe9cd00a0af1022b4"
server_key_iv: "b0845b6fd4340c5b5f63ce217ffaae50"
server_key_version: "kuduclusterkey@0"
I20260621 02:47:15.783170 22952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:15.811060 22952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:15.816406 22952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:15.816710 22952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:15.817983 22952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:47:15.819962 22952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:47:15.820173 22952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260621 02:47:15.820330 22952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:47:15.820438 22952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 21 02:47:15 dist-test-slave-hvgn krb5kdc[21740](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1782010035, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:15.883692 22952 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:44573
I20260621 02:47:15.883765 23091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:44573 every 8 connection(s)
I20260621 02:47:15.885627 22952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:47:15.886689 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 22952
I20260621 02:47:15.887548 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.166.20:32969/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:43219
--local_ip_for_outbound_sockets=127.9.166.3
--tserver_master_addrs=127.9.166.62:36885
--webserver_port=42399
--webserver_interface=127.9.166.3
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:37703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.166.20:32969/kms with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:47:15 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010035, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:15.925704 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:15.900407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:43845 (local address 127.9.166.62:36885)
0621 02:47:15.902372 (+ 1965us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:15.902379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:15.906430 (+ 4051us) server_negotiation.cc:407] Connection header received
0621 02:47:15.906506 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:15.906512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:15.906599 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:15.906715 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:15.909331 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:15.910507 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:15.913015 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:15.913675 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:15.913776 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:15.918319 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:15.918326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:15.918332 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:15.918372 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:15.921925 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:15.922863 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:15.922870 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:15.922874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:15.923033 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:15.923629 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:15.923635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:15.923638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:15.923975 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:15.924123 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:15.925099 (+ 976us) server_negotiation.cc:299] Negotiation successful
0621 02:47:15.925322 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1833,"thread_start_us":1721,"threads_started":1}
I20260621 02:47:15.928320 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:15.900747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:36885 (local address 127.9.166.2:43845)
0621 02:47:15.903494 (+ 2747us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:15.903558 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:15.905996 (+ 2438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:15.907592 (+ 1596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:15.907604 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:15.908251 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:15.909111 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:15.909137 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:15.910717 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:15.910724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:15.912828 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:15.912844 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:15.914136 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:15.914144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:15.914428 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:15.915236 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:15.915270 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:15.918057 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:15.922113 (+ 4056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:15.922122 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:15.922139 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:15.922720 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:15.923182 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:15.923187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:15.923190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:15.923498 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:15.924222 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:15.924231 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:15.924649 (+ 418us) client_negotiation.cc:769] Sending connection context
0621 02:47:15.926264 (+ 1615us) client_negotiation.cc:240] Negotiation successful
0621 02:47:15.926831 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2421,"thread_start_us":2212,"threads_started":1}
I20260621 02:47:15.930198 23092 heartbeater.cc:347] Connected to a master server at 127.9.166.62:36885
I20260621 02:47:15.930502 23092 heartbeater.cc:464] Registering TS with master...
I20260621 02:47:15.931480 23092 heartbeater.cc:511] Master 127.9.166.62:36885 requested a full tablet report, sending...
I20260621 02:47:15.933511 22751 ts_manager.cc:194] Registered new tserver with Master: 841ce961c8b843f3a40579c70c1a0b48 (127.9.166.2:44573)
I20260621 02:47:15.935707 22751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:43845
W20260621 02:47:16.312566 23093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:16.312942 23093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:16.313122 23093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:16.324717 23093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:16.324892 23093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:16.325091 23093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:47:16.333422 22948 heartbeater.cc:464] Registering TS with master...
I20260621 02:47:16.333840 22948 heartbeater.cc:511] Master 127.9.166.62:36885 requested a full tablet report, sending...
I20260621 02:47:16.335063 22751 ts_manager.cc:194] Registered new tserver with Master: cdab9cda14cf4edd9b36302e1c140181 (127.9.166.1:38297)
I20260621 02:47:16.336499 22751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:40205
I20260621 02:47:16.340145 23093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:37703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.166.20:32969/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:43219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=42399
--tserver_master_addrs=127.9.166.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:16.341589 23093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:16.343170 23093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 21 02:47:16 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010035, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
W20260621 02:47:16.360680 23103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:16.366598 23102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:16.405362 23105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:16.406152 23093 server_base.cc:1034] running on GCE node
I20260621 02:47:16.407246 23093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:16.414777 23093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:16.416311 23093 hybrid_clock.cc:648] HybridClock initialized: now 1782010036416238 us; error 61 us; skew 500 ppm
I20260621 02:47:16.416775 23093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:16 dist-test-slave-hvgn krb5kdc[21740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010036, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:16.424011 23093 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:47:16.451452 23093 webserver.cc:460] Webserver started at http://127.9.166.3:42399/ using document root <none> and password file <none>
I20260621 02:47:16.452594 23093 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:16.452832 23093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 21 02:47:16 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010036, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
I20260621 02:47:16.513123 23093 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.006s sys 0.009s
I20260621 02:47:16.523098 23113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:16.525046 23093 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.009s sys 0.000s
I20260621 02:47:16.525305 23093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "0896084bbfb34282b5071fc849b9b00c"
format_stamp: "Formatted at 2026-06-21 02:47:13 on dist-test-slave-hvgn"
server_key: "f99def0660dd89d276145ee507bc1c53"
server_key_iv: "fd99fb32a7abb1c11f8d8ed35da214c8"
server_key_version: "kuduclusterkey@0"
I20260621 02:47:16.526912 23093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:16.577517 23093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:16.583247 23093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:16.583600 23093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:16.584935 23093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:47:16.586823 23093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:47:16.586992 23093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:16.587148 23093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:47:16.587266 23093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:16.639711 23093 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:43219
I20260621 02:47:16.639768 23225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:43219 every 8 connection(s)
I20260621 02:47:16.641273 23093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:47:16.650981 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 23093
Jun 21 02:47:16 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010036, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:16.723881 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:16.652908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53266 (local address 127.9.166.62:36885)
0621 02:47:16.696230 (+ 43322us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:16.696237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:16.696259 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:47:16.696330 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:16.696338 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:16.696473 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:16.696624 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:16.703422 (+ 6798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:16.704564 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:16.706699 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:16.707329 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:16.708401 (+ 1072us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:16.714820 (+ 6419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:16.714826 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:16.714831 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:16.714874 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:16.719196 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:16.720321 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:16.720329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:16.720333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:16.720499 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:16.721010 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:16.721019 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:16.721026 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:16.721393 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:16.721680 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:16.723181 (+ 1501us) server_negotiation.cc:299] Negotiation successful
0621 02:47:16.723457 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43178,"thread_start_us":40804,"threads_started":1}
I20260621 02:47:16.729562 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:16.646117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:60565 (local address 127.9.166.62:36885)
0621 02:47:16.692856 (+ 46739us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:16.692863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:16.702937 (+ 10074us) server_negotiation.cc:407] Connection header received
0621 02:47:16.703001 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:16.703006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:16.703092 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:16.703187 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:16.713168 (+ 9981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:16.714371 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:16.716892 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:16.717593 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:16.717695 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:16.722263 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:16.722269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:16.722274 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:16.722309 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:16.725860 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:16.726748 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:16.726753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:16.726756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:16.726896 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:16.727482 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:16.727488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:16.727491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:16.727800 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:16.727947 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:16.728997 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0621 02:47:16.729251 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46572,"thread_start_us":50639,"threads_started":1}
I20260621 02:47:16.731520 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:16.645357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:36885 (local address 127.9.166.3:60565)
0621 02:47:16.699892 (+ 54535us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:16.699965 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:16.702473 (+ 2508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:16.708902 (+ 6429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:16.708916 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:16.709604 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:16.710510 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:16.710534 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:16.714578 (+ 4044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:16.714584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:16.716717 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:16.716732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:16.718166 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:16.718176 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:16.718485 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:16.719233 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:16.719263 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:16.721984 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:16.726016 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:16.726025 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:16.726043 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:16.726613 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:16.727039 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:16.727043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:16.727046 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:16.727290 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:16.728089 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:16.728098 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:16.728528 (+ 430us) client_negotiation.cc:769] Sending connection context
0621 02:47:16.729476 (+ 948us) client_negotiation.cc:240] Negotiation successful
0621 02:47:16.730046 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54160,"thread_start_us":61912,"threads_started":1}
I20260621 02:47:16.733529 23226 heartbeater.cc:347] Connected to a master server at 127.9.166.62:36885
I20260621 02:47:16.733845 23226 heartbeater.cc:464] Registering TS with master...
I20260621 02:47:16.734727 23226 heartbeater.cc:511] Master 127.9.166.62:36885 requested a full tablet report, sending...
I20260621 02:47:16.736359 22751 ts_manager.cc:194] Registered new tserver with Master: 0896084bbfb34282b5071fc849b9b00c (127.9.166.3:43219)
I20260621 02:47:16.737444 22751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:60565
I20260621 02:47:16.739780 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 21 02:47:16 dist-test-slave-hvgn krb5kdc[21740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010035, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for HTTP/127.9.166.20@KRBTEST.COM
I20260621 02:47:16.780355 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:16.749336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53268 (local address 127.9.166.62:36885)
0621 02:47:16.749680 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:16.749687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:16.752162 (+ 2475us) server_negotiation.cc:407] Connection header received
0621 02:47:16.752336 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:16.752342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:16.752437 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:16.752551 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:16.757293 (+ 4742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:16.758222 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:16.760364 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:16.761036 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:16.761134 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:16.773163 (+ 12029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:16.773171 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:16.773176 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:16.773227 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:16.776936 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:16.777915 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:16.777921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:16.777925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:16.778072 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:16.778520 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:16.778525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:16.778528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:16.778892 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:16.779119 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:16.779738 (+ 619us) server_negotiation.cc:299] Negotiation successful
0621 02:47:16.779988 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221}
I20260621 02:47:16.787758 22751 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:47:16.801163 22751 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:47:16.873555 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:16.841400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:43219 (local address 127.0.0.1:51630)
0621 02:47:16.850822 (+ 9422us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:16.850866 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:16.851074 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:16.859138 (+ 8064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:16.859149 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:16.859193 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:16.859751 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:16.859769 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:16.865066 (+ 5297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:16.865073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:16.867510 (+ 2437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:16.867532 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:16.872246 (+ 4714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:16.872255 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:16.872549 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:16.872631 (+ 82us) client_negotiation.cc:769] Sending connection context
0621 02:47:16.872768 (+ 137us) client_negotiation.cc:240] Negotiation successful
0621 02:47:16.873058 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9142,"thread_start_us":8975,"threads_started":1}
I20260621 02:47:16.875058 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:16.841555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.9.166.3:43219)
0621 02:47:16.858091 (+ 16536us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:16.858104 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:16.858148 (+ 44us) server_negotiation.cc:407] Connection header received
0621 02:47:16.858281 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:16.858293 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:16.858624 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:16.859988 (+ 1364us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:16.860325 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:16.862260 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:16.867768 (+ 5508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:16.869775 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:16.874178 (+ 4403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:16.874446 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:16.874539 (+ 93us) server_negotiation.cc:299] Negotiation successful
0621 02:47:16.874714 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16300,"thread_start_us":17259,"threads_started":1}
I20260621 02:47:16.891741 23161 tablet_service.cc:1467] Processing CreateTablet for tablet 9595704231fd4074ad0a78b624f8fb6a (DEFAULT_TABLE table=test-table [id=c5604490cc4242fdb03749a04c46f0f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:47:16.893388 23161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9595704231fd4074ad0a78b624f8fb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:47:16.899219 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:16.844524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:44573 (local address 127.0.0.1:58176)
0621 02:47:16.873856 (+ 29332us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:16.873875 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:16.874007 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:16.881952 (+ 7945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:16.881960 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:16.881991 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:16.882172 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:16.882183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:16.894635 (+ 12452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:16.894640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:16.896543 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:16.896558 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:16.898398 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:16.898405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:16.898612 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:16.898646 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 02:47:16.898715 (+ 69us) client_negotiation.cc:240] Negotiation successful
0621 02:47:16.898872 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29233,"mutex_wait_us":1,"thread_start_us":53938,"threads_started":1}
I20260621 02:47:16.900279 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:16.845188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58176 (local address 127.9.166.2:44573)
0621 02:47:16.880979 (+ 35791us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:16.880989 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:16.881021 (+ 32us) server_negotiation.cc:407] Connection header received
0621 02:47:16.881130 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:16.881138 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:16.881452 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:16.882356 (+ 904us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:16.890867 (+ 8511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:16.892617 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:16.896765 (+ 4148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:16.898204 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:16.899501 (+ 1297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:16.899733 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:16.899808 (+ 75us) server_negotiation.cc:299] Negotiation successful
0621 02:47:16.899970 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35608,"thread_start_us":36424,"threads_started":1}
I20260621 02:47:16.903425 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:16.857606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:38297 (local address 127.0.0.1:51332)
0621 02:47:16.886139 (+ 28533us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:16.886166 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:16.886296 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:16.893931 (+ 7635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:16.893936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:16.893968 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:16.894377 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:16.894394 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:16.896440 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:16.896446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:16.898702 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:16.898715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:16.902611 (+ 3896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:16.902618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:16.902787 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:16.902823 (+ 36us) client_negotiation.cc:769] Sending connection context
0621 02:47:16.902929 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 02:47:16.903056 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28397,"mutex_wait_us":188,"thread_start_us":28250,"threads_started":1}
I20260621 02:47:16.904121 23027 tablet_service.cc:1467] Processing CreateTablet for tablet 9595704231fd4074ad0a78b624f8fb6a (DEFAULT_TABLE table=test-table [id=c5604490cc4242fdb03749a04c46f0f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:47:16.905691 23027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9595704231fd4074ad0a78b624f8fb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:47:16.909044 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:16.851714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51332 (local address 127.9.166.1:38297)
0621 02:47:16.893233 (+ 41519us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:16.893242 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:16.893272 (+ 30us) server_negotiation.cc:407] Connection header received
0621 02:47:16.893387 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:16.893395 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:16.893688 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:16.893904 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:16.894635 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:16.896253 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:16.898852 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:16.902110 (+ 3258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:16.902243 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:16.902464 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:16.908491 (+ 6027us) server_negotiation.cc:299] Negotiation successful
0621 02:47:16.908691 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41334,"thread_start_us":51671,"threads_started":1}
I20260621 02:47:16.913187 22883 tablet_service.cc:1467] Processing CreateTablet for tablet 9595704231fd4074ad0a78b624f8fb6a (DEFAULT_TABLE table=test-table [id=c5604490cc4242fdb03749a04c46f0f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:47:16.914722 22883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9595704231fd4074ad0a78b624f8fb6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:47:16.928258 23248 tablet_bootstrap.cc:492] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48: Bootstrap starting.
I20260621 02:47:16.932117 23248 tablet_bootstrap.cc:654] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48: Neither blocks nor log segments found. Creating new log.
I20260621 02:47:16.934861 23248 log.cc:824] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:16.937064 23249 tablet_bootstrap.cc:492] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c: Bootstrap starting.
I20260621 02:47:16.940398 23092 heartbeater.cc:503] Master 127.9.166.62:36885 was elected leader, sending a full tablet report...
I20260621 02:47:16.942879 23251 tablet_bootstrap.cc:492] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181: Bootstrap starting.
I20260621 02:47:16.944255 23248 tablet_bootstrap.cc:492] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48: No bootstrap required, opened a new log
I20260621 02:47:16.944546 23248 ts_tablet_manager.cc:1397] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.008s
I20260621 02:47:16.946764 23251 tablet_bootstrap.cc:654] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181: Neither blocks nor log segments found. Creating new log.
I20260621 02:47:16.948248 23251 log.cc:824] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:16.951539 23249 tablet_bootstrap.cc:654] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c: Neither blocks nor log segments found. Creating new log.
I20260621 02:47:16.953250 23249 log.cc:824] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:16.954900 23248 raft_consensus.cc:348] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0896084bbfb34282b5071fc849b9b00c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43219 } } peers { permanent_uuid: "cdab9cda14cf4edd9b36302e1c140181" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 38297 } } peers { permanent_uuid: "841ce961c8b843f3a40579c70c1a0b48" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44573 } }
I20260621 02:47:16.955336 23248 raft_consensus.cc:374] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:47:16.955554 23248 raft_consensus.cc:729] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841ce961c8b843f3a40579c70c1a0b48, State: Initialized, Role: FOLLOWER
I20260621 02:47:16.956583 23248 consensus_queue.cc:260] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0896084bbfb34282b5071fc849b9b00c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43219 } } peers { permanent_uuid: "cdab9cda14cf4edd9b36302e1c140181" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 38297 } } peers { permanent_uuid: "841ce961c8b843f3a40579c70c1a0b48" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44573 } }
I20260621 02:47:16.957834 23251 tablet_bootstrap.cc:492] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181: No bootstrap required, opened a new log
I20260621 02:47:16.958129 23251 ts_tablet_manager.cc:1397] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260621 02:47:16.964919 23248 ts_tablet_manager.cc:1428] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48: Time spent starting tablet: real 0.020s user 0.006s sys 0.013s
I20260621 02:47:16.965864 23249 tablet_bootstrap.cc:492] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c: No bootstrap required, opened a new log
I20260621 02:47:16.966169 23249 ts_tablet_manager.cc:1397] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.005s
I20260621 02:47:16.982018 23251 raft_consensus.cc:348] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0896084bbfb34282b5071fc849b9b00c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43219 } } peers { permanent_uuid: "cdab9cda14cf4edd9b36302e1c140181" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 38297 } } peers { permanent_uuid: "841ce961c8b843f3a40579c70c1a0b48" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44573 } }
I20260621 02:47:16.982549 23251 raft_consensus.cc:374] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:47:16.982666 23251 raft_consensus.cc:729] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdab9cda14cf4edd9b36302e1c140181, State: Initialized, Role: FOLLOWER
I20260621 02:47:16.983489 23251 consensus_queue.cc:260] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0896084bbfb34282b5071fc849b9b00c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43219 } } peers { permanent_uuid: "cdab9cda14cf4edd9b36302e1c140181" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 38297 } } peers { permanent_uuid: "841ce961c8b843f3a40579c70c1a0b48" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44573 } }
I20260621 02:47:16.986265 23249 raft_consensus.cc:348] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0896084bbfb34282b5071fc849b9b00c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43219 } } peers { permanent_uuid: "cdab9cda14cf4edd9b36302e1c140181" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 38297 } } peers { permanent_uuid: "841ce961c8b843f3a40579c70c1a0b48" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44573 } }
I20260621 02:47:16.986732 23249 raft_consensus.cc:374] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:47:16.986837 23249 raft_consensus.cc:729] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0896084bbfb34282b5071fc849b9b00c, State: Initialized, Role: FOLLOWER
I20260621 02:47:16.989212 23249 consensus_queue.cc:260] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0896084bbfb34282b5071fc849b9b00c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43219 } } peers { permanent_uuid: "cdab9cda14cf4edd9b36302e1c140181" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 38297 } } peers { permanent_uuid: "841ce961c8b843f3a40579c70c1a0b48" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44573 } }
I20260621 02:47:16.990754 23226 heartbeater.cc:503] Master 127.9.166.62:36885 was elected leader, sending a full tablet report...
I20260621 02:47:16.991966 23249 ts_tablet_manager.cc:1428] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c: Time spent starting tablet: real 0.026s user 0.010s sys 0.006s
I20260621 02:47:16.995846 23251 ts_tablet_manager.cc:1428] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181: Time spent starting tablet: real 0.038s user 0.015s sys 0.004s
I20260621 02:47:16.996212 22948 heartbeater.cc:503] Master 127.9.166.62:36885 was elected leader, sending a full tablet report...
W20260621 02:47:17.012531 22949 tablet.cc:2369] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:47:17.156399 23094 tablet.cc:2369] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:47:17.211762 23227 tablet.cc:2369] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:47:17.396670 23256 raft_consensus.cc:482] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:47:17.396987 23256 raft_consensus.cc:504] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0896084bbfb34282b5071fc849b9b00c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43219 } } peers { permanent_uuid: "cdab9cda14cf4edd9b36302e1c140181" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 38297 } } peers { permanent_uuid: "841ce961c8b843f3a40579c70c1a0b48" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44573 } }
I20260621 02:47:17.398545 23256 leader_election.cc:288] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdab9cda14cf4edd9b36302e1c140181 (127.9.166.1:38297), 841ce961c8b843f3a40579c70c1a0b48 (127.9.166.2:44573)
I20260621 02:47:17.399765 23254 raft_consensus.cc:482] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:47:17.400125 23254 raft_consensus.cc:504] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0896084bbfb34282b5071fc849b9b00c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43219 } } peers { permanent_uuid: "cdab9cda14cf4edd9b36302e1c140181" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 38297 } } peers { permanent_uuid: "841ce961c8b843f3a40579c70c1a0b48" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44573 } }
I20260621 02:47:17.402099 23254 leader_election.cc:288] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0896084bbfb34282b5071fc849b9b00c (127.9.166.3:43219), cdab9cda14cf4edd9b36302e1c140181 (127.9.166.1:38297)
I20260621 02:47:17.408635 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:17.399081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:60515 (local address 127.9.166.2:44573)
0621 02:47:17.399288 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:17.399297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:17.400089 (+ 792us) server_negotiation.cc:407] Connection header received
0621 02:47:17.400348 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:17.400351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:17.400465 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:17.400556 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:17.401135 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.402017 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.405349 (+ 3332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.407119 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.407233 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:17.407330 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:17.408084 (+ 754us) server_negotiation.cc:299] Negotiation successful
0621 02:47:17.408271 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 02:47:17.409017 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:17.399060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:40293 (local address 127.9.166.1:38297)
0621 02:47:17.399304 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:17.399309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:17.399663 (+ 354us) server_negotiation.cc:407] Connection header received
0621 02:47:17.399758 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:17.399765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:17.399858 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:17.399959 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:17.400483 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.401409 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.404354 (+ 2945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.406080 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.406233 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:17.406395 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:17.408492 (+ 2097us) server_negotiation.cc:299] Negotiation successful
0621 02:47:17.408690 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:47:17.409359 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:17.398818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:38297 (local address 127.9.166.3:40293)
0621 02:47:17.399312 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:17.399331 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:17.399533 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:17.399985 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:17.399990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:17.400021 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:17.400314 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:17.400327 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.401565 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.401571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:17.404108 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:17.404127 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.406811 (+ 2684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.406820 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:17.407088 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:17.407140 (+ 52us) client_negotiation.cc:769] Sending connection context
0621 02:47:17.407251 (+ 111us) client_negotiation.cc:240] Negotiation successful
0621 02:47:17.407471 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":257,"threads_started":1}
I20260621 02:47:17.410420 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:17.398818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:44573 (local address 127.9.166.3:60515)
0621 02:47:17.399995 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:17.400018 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:17.400184 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:17.400583 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:17.400587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:17.400612 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:17.400950 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:17.400961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.402190 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.402200 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:17.405102 (+ 2902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:17.405122 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.407546 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.407555 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:17.407850 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:17.407902 (+ 52us) client_negotiation.cc:769] Sending connection context
0621 02:47:17.409094 (+ 1192us) client_negotiation.cc:240] Negotiation successful
0621 02:47:17.409295 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":499,"thread_start_us":405,"threads_started":1}
I20260621 02:47:17.411291 22903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9595704231fd4074ad0a78b624f8fb6a" candidate_uuid: "0896084bbfb34282b5071fc849b9b00c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdab9cda14cf4edd9b36302e1c140181" is_pre_election: true
I20260621 02:47:17.411705 23047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9595704231fd4074ad0a78b624f8fb6a" candidate_uuid: "0896084bbfb34282b5071fc849b9b00c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841ce961c8b843f3a40579c70c1a0b48" is_pre_election: true
I20260621 02:47:17.411938 22903 raft_consensus.cc:2454] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0896084bbfb34282b5071fc849b9b00c in term 0.
I20260621 02:47:17.412163 23047 raft_consensus.cc:2454] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0896084bbfb34282b5071fc849b9b00c in term 0.
I20260621 02:47:17.412822 23117 leader_election.cc:302] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0896084bbfb34282b5071fc849b9b00c, 841ce961c8b843f3a40579c70c1a0b48; no voters:
I20260621 02:47:17.413259 23256 raft_consensus.cc:2781] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:47:17.413440 23256 raft_consensus.cc:482] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:47:17.413587 23256 raft_consensus.cc:3037] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:47:17.416604 23256 raft_consensus.cc:504] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0896084bbfb34282b5071fc849b9b00c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43219 } } peers { permanent_uuid: "cdab9cda14cf4edd9b36302e1c140181" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 38297 } } peers { permanent_uuid: "841ce961c8b843f3a40579c70c1a0b48" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44573 } }
I20260621 02:47:17.417565 23256 leader_election.cc:288] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [CANDIDATE]: Term 1 election: Requested vote from peers cdab9cda14cf4edd9b36302e1c140181 (127.9.166.1:38297), 841ce961c8b843f3a40579c70c1a0b48 (127.9.166.2:44573)
I20260621 02:47:17.417671 22903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9595704231fd4074ad0a78b624f8fb6a" candidate_uuid: "0896084bbfb34282b5071fc849b9b00c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdab9cda14cf4edd9b36302e1c140181"
I20260621 02:47:17.417726 23047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9595704231fd4074ad0a78b624f8fb6a" candidate_uuid: "0896084bbfb34282b5071fc849b9b00c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841ce961c8b843f3a40579c70c1a0b48"
I20260621 02:47:17.417922 23047 raft_consensus.cc:3037] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:47:17.417912 22903 raft_consensus.cc:3037] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:47:17.418538 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:17.402307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:43219 (local address 127.9.166.2:42141)
0621 02:47:17.409864 (+ 7557us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:17.409885 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:17.410002 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:17.410841 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:17.410849 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:17.410888 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:17.411456 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:17.411473 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.413535 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.413542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:17.415881 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:17.415898 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.417661 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.417669 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:17.417852 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:17.417889 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 02:47:17.417967 (+ 78us) client_negotiation.cc:240] Negotiation successful
0621 02:47:17.418101 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7428,"thread_start_us":7272,"threads_started":1}
I20260621 02:47:17.418977 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:17.402482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:42141 (local address 127.9.166.3:43219)
0621 02:47:17.410476 (+ 7994us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:17.410481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:17.410498 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:47:17.410564 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:17.410570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:17.410661 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:17.411624 (+ 963us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:17.411909 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.413365 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.416039 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.417407 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.418152 (+ 745us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:17.418285 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:17.418357 (+ 72us) server_negotiation.cc:299] Negotiation successful
0621 02:47:17.418500 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7879,"thread_start_us":7749,"threads_started":1}
I20260621 02:47:17.419778 23181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9595704231fd4074ad0a78b624f8fb6a" candidate_uuid: "841ce961c8b843f3a40579c70c1a0b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0896084bbfb34282b5071fc849b9b00c" is_pre_election: true
I20260621 02:47:17.420260 23181 raft_consensus.cc:2379] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 841ce961c8b843f3a40579c70c1a0b48 in current term 1: Already voted for candidate 0896084bbfb34282b5071fc849b9b00c in this term.
I20260621 02:47:17.421146 22903 raft_consensus.cc:2454] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0896084bbfb34282b5071fc849b9b00c in term 1.
I20260621 02:47:17.421305 23047 raft_consensus.cc:2454] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0896084bbfb34282b5071fc849b9b00c in term 1.
I20260621 02:47:17.421546 23114 leader_election.cc:302] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0896084bbfb34282b5071fc849b9b00c, cdab9cda14cf4edd9b36302e1c140181; no voters:
I20260621 02:47:17.421844 23256 raft_consensus.cc:2781] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:47:17.422355 23256 raft_consensus.cc:686] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [term 1 LEADER]: Becoming Leader. State: Replica: 0896084bbfb34282b5071fc849b9b00c, State: Running, Role: LEADER
I20260621 02:47:17.422626 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:17.410147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:48195 (local address 127.9.166.1:38297)
0621 02:47:17.410323 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:17.410329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:17.413539 (+ 3210us) server_negotiation.cc:407] Connection header received
0621 02:47:17.413788 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:17.413795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:17.413861 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:17.413957 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:17.414676 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.415368 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.419620 (+ 4252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.421018 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.421137 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:17.421283 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:17.422254 (+ 971us) server_negotiation.cc:299] Negotiation successful
0621 02:47:17.422383 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:47:17.422670 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:17.410469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:38297 (local address 127.9.166.2:48195)
0621 02:47:17.413450 (+ 2981us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:17.413468 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:17.413643 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:17.413983 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:17.413987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:17.414012 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:17.414527 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:17.414537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.415543 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.415548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:17.417430 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:17.417445 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.421973 (+ 4528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.421980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:17.422105 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:17.422126 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 02:47:17.422231 (+ 105us) client_negotiation.cc:240] Negotiation successful
0621 02:47:17.422352 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2853,"thread_start_us":3680,"threads_started":1}
I20260621 02:47:17.423045 23256 consensus_queue.cc:237] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0896084bbfb34282b5071fc849b9b00c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43219 } } peers { permanent_uuid: "cdab9cda14cf4edd9b36302e1c140181" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 38297 } } peers { permanent_uuid: "841ce961c8b843f3a40579c70c1a0b48" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44573 } }
I20260621 02:47:17.423194 22903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9595704231fd4074ad0a78b624f8fb6a" candidate_uuid: "841ce961c8b843f3a40579c70c1a0b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdab9cda14cf4edd9b36302e1c140181" is_pre_election: true
I20260621 02:47:17.423493 22903 raft_consensus.cc:2379] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 841ce961c8b843f3a40579c70c1a0b48 in current term 1: Already voted for candidate 0896084bbfb34282b5071fc849b9b00c in this term.
I20260621 02:47:17.424062 22980 leader_election.cc:302] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 841ce961c8b843f3a40579c70c1a0b48; no voters: 0896084bbfb34282b5071fc849b9b00c, cdab9cda14cf4edd9b36302e1c140181
I20260621 02:47:17.424443 23254 raft_consensus.cc:2726] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260621 02:47:17.428095 22751 catalog_manager.cc:5495] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0896084bbfb34282b5071fc849b9b00c (127.9.166.3). New cstate: current_term: 1 leader_uuid: "0896084bbfb34282b5071fc849b9b00c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0896084bbfb34282b5071fc849b9b00c" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdab9cda14cf4edd9b36302e1c140181" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 38297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "841ce961c8b843f3a40579c70c1a0b48" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44573 } health_report { overall_health: UNKNOWN } } }
I20260621 02:47:17.482528 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:17.477001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51646 (local address 127.9.166.3:43219)
0621 02:47:17.477171 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:17.477175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:17.477571 (+ 396us) server_negotiation.cc:407] Connection header received
0621 02:47:17.477782 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:17.477786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:17.477860 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:17.477987 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:47:17.478316 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.479041 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.480499 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.480918 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.481017 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:17.481491 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:47:17.482066 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:47:17.482189 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:17.482265 (+ 76us) server_negotiation.cc:299] Negotiation successful
0621 02:47:17.482330 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 02:47:17.490311 22903 raft_consensus.cc:1261] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181 [term 1 FOLLOWER]: Refusing update from remote peer 0896084bbfb34282b5071fc849b9b00c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:47:17.490345 23047 raft_consensus.cc:1261] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48 [term 1 FOLLOWER]: Refusing update from remote peer 0896084bbfb34282b5071fc849b9b00c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:47:17.491438 23256 consensus_queue.cc:1035] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [LEADER]: Connected to new peer: Peer: permanent_uuid: "841ce961c8b843f3a40579c70c1a0b48" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 44573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:47:17.491755 23269 consensus_queue.cc:1035] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdab9cda14cf4edd9b36302e1c140181" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 38297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:47:17.503325 23276 mvcc.cc:205] Tried to move back new op lower bound from 7299113113550872576 to 7299113113289490432. Current Snapshot: MvccSnapshot[applied={T|T < 7299113113550872576}]
I20260621 02:47:17.525067 22750 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:47:17.525346 22750 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:47:17.528434 22750 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21d2ab7f3918407fb103e4c02d567ee5: Sending DeleteTablet for 3 replicas of tablet 9595704231fd4074ad0a78b624f8fb6a
I20260621 02:47:17.529593 23027 tablet_service.cc:1514] Processing DeleteTablet for tablet 9595704231fd4074ad0a78b624f8fb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:47:17 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:58176
I20260621 02:47:17.529872 23161 tablet_service.cc:1514] Processing DeleteTablet for tablet 9595704231fd4074ad0a78b624f8fb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:47:17 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:51630
I20260621 02:47:17.530277 22883 tablet_service.cc:1514] Processing DeleteTablet for tablet 9595704231fd4074ad0a78b624f8fb6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:47:17 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:51332
I20260621 02:47:17.531158 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 22788
I20260621 02:47:17.536599 23287 tablet_replica.cc:330] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c: stopping tablet replica
I20260621 02:47:17.536818 23286 tablet_replica.cc:330] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48: stopping tablet replica
I20260621 02:47:17.537233 23286 raft_consensus.cc:2229] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:47:17.537732 23286 raft_consensus.cc:2258] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:47:17.538452 23288 tablet_replica.cc:330] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181: stopping tablet replica
I20260621 02:47:17.537731 23287 raft_consensus.cc:2229] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:47:17.538826 23288 raft_consensus.cc:2229] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:47:17.539105 23287 raft_consensus.cc:2258] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:47:17.539286 23288 raft_consensus.cc:2258] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:47:17.541270 23288 ts_tablet_manager.cc:1905] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:47:17.542562 23286 ts_tablet_manager.cc:1905] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:47:17.549070 23287 ts_tablet_manager.cc:1905] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:47:17.549903 23286 ts_tablet_manager.cc:1918] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:47:17.550097 23286 log.cc:1196] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/9595704231fd4074ad0a78b624f8fb6a
I20260621 02:47:17.550606 23286 ts_tablet_manager.cc:1939] T 9595704231fd4074ad0a78b624f8fb6a P 841ce961c8b843f3a40579c70c1a0b48: Deleting consensus metadata
I20260621 02:47:17.551285 23288 ts_tablet_manager.cc:1918] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:47:17.551625 23288 log.cc:1196] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/9595704231fd4074ad0a78b624f8fb6a
I20260621 02:47:17.552224 23288 ts_tablet_manager.cc:1939] T 9595704231fd4074ad0a78b624f8fb6a P cdab9cda14cf4edd9b36302e1c140181: Deleting consensus metadata
I20260621 02:47:17.552527 22738 catalog_manager.cc:4841] TS 841ce961c8b843f3a40579c70c1a0b48 (127.9.166.2:44573): tablet 9595704231fd4074ad0a78b624f8fb6a (table test-table [id=c5604490cc4242fdb03749a04c46f0f3]) successfully deleted
I20260621 02:47:17.553948 22735 catalog_manager.cc:4841] TS cdab9cda14cf4edd9b36302e1c140181 (127.9.166.1:38297): tablet 9595704231fd4074ad0a78b624f8fb6a (table test-table [id=c5604490cc4242fdb03749a04c46f0f3]) successfully deleted
I20260621 02:47:17.555351 23287 ts_tablet_manager.cc:1918] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:47:17.555546 23287 log.cc:1196] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/9595704231fd4074ad0a78b624f8fb6a
I20260621 02:47:17.555979 23287 ts_tablet_manager.cc:1939] T 9595704231fd4074ad0a78b624f8fb6a P 0896084bbfb34282b5071fc849b9b00c: Deleting consensus metadata
I20260621 02:47:17.557277 22735 catalog_manager.cc:4841] TS 0896084bbfb34282b5071fc849b9b00c (127.9.166.3:43219): tablet 9595704231fd4074ad0a78b624f8fb6a (table test-table [id=c5604490cc4242fdb03749a04c46f0f3]) successfully deleted
I20260621 02:47:17.560568 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:17.532941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51342 (local address 127.9.166.1:38297)
0621 02:47:17.533145 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:17.533154 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:17.536006 (+ 2852us) server_negotiation.cc:407] Connection header received
0621 02:47:17.536181 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:17.536190 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:17.536286 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:17.536398 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:17.538561 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.539621 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.541010 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.541440 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.541575 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:17.544618 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:17.544639 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:17.544660 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:17.544697 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:17.557281 (+ 12584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:17.558209 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:17.558220 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:17.558224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:17.558391 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:17.558848 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:17.558854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:17.558858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:17.559331 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:17.559703 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:17.559981 (+ 278us) server_negotiation.cc:299] Negotiation successful
0621 02:47:17.560210 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:47:17.561229 22943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:17.656085 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 22788
I20260621 02:47:17.673314 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 22952
I20260621 02:47:17.687291 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:17.674139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58184 (local address 127.9.166.2:44573)
0621 02:47:17.674318 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:17.674324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:17.674438 (+ 114us) server_negotiation.cc:407] Connection header received
0621 02:47:17.674648 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:17.674653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:17.674764 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:17.674885 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:17.676236 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.676958 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.678269 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.678698 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.678848 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:17.681395 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:17.681418 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:17.681432 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:17.681469 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:17.684515 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:17.685211 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:17.685219 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:17.685221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:17.685349 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:17.685717 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:17.685723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:17.685726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:17.686126 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:17.686432 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:17.686731 (+ 299us) server_negotiation.cc:299] Negotiation successful
0621 02:47:17.686955 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:47:17.688185 23087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:17.778614 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 22952
I20260621 02:47:17.795727 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 23093
I20260621 02:47:17.811244 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:17.796511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51648 (local address 127.9.166.3:43219)
0621 02:47:17.796731 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:17.796739 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:17.796760 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:47:17.796914 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:17.796920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:17.797030 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:17.797151 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:17.798400 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.799610 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.800982 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:17.801679 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:17.801806 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:17.804362 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:17.804376 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:17.804393 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:17.804433 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:17.808423 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:17.809117 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:17.809129 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:17.809131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:17.809288 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:17.809667 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:17.809674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:17.809682 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:17.810049 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:17.810403 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:17.810658 (+ 255us) server_negotiation.cc:299] Negotiation successful
0621 02:47:17.810873 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260621 02:47:17.812063 23221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:17.935595 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 23093
I20260621 02:47:17.954613 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 22718
I20260621 02:47:17.955826 22781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:18.066210 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 22718
I20260621 02:47:18.090317 9880 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260621 02:47:18.435410 9880 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260621 02:47:18.435654 9880 mini_ranger.cc:66] Stopping Ranger...
I20260621 02:47:18.780333 9880 mini_ranger.cc:68] Stopped Ranger
2026-06-21 02:47:18.780 UTC [21770] LOG: received smart shutdown request
2026-06-21 02:47:18.783 UTC [21770] LOG: background worker "logical replication launcher" (PID 21779) exited with exit code 1
2026-06-21 02:47:18.783 UTC [21774] LOG: shutting down
2026-06-21 02:47:18.938 UTC [21770] LOG: database system is shut down
2026-06-21T02:47:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51651 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:47:19 dist-test-slave-hvgn krb5kdc[23302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:47:19 dist-test-slave-hvgn krb5kdc[23302](info): set up 2 sockets
Jun 21 02:47:19 dist-test-slave-hvgn krb5kdc[23302](info): commencing operation
krb5kdc: starting...
W20260621 02:47:20.195955 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:47:20 dist-test-slave-hvgn krb5kdc[23302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:47:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:47:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:47:20.416891 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:34823
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:34823,127.9.166.61:38133,127.9.166.60:40503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:47:20.732757 23315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:20.733043 23315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:20.733210 23315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:20.740371 23315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:47:20.740473 23315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:20.740577 23315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:20.740677 23315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:47:20.740760 23315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:47:20.750340 23315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:34823,127.9.166.61:38133,127.9.166.60:40503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:34823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:20.751681 23315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:20.752972 23315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:20.764407 23323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:20.764384 23320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:20.764504 23321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:20.765504 23315 server_base.cc:1034] running on GCE node
I20260621 02:47:20.766281 23315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:20.767725 23315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:20.768946 23315 hybrid_clock.cc:648] HybridClock initialized: now 1782010040768900 us; error 43 us; skew 500 ppm
I20260621 02:47:20.769292 23315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:20 dist-test-slave-hvgn krb5kdc[23302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010040, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:20.772460 23315 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:47:20.773845 23315 webserver.cc:460] Webserver started at http://127.9.166.62:42253/ using document root <none> and password file <none>
I20260621 02:47:20.774528 23315 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:20.774670 23315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:20.774969 23315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:47:20.777922 23315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "c2f30dd97473432ab3340fddd1d93853"
format_stamp: "Formatted at 2026-06-21 02:47:20 on dist-test-slave-hvgn"
server_key: "5505999ac55ad2a50be2425ba75efe12"
server_key_iv: "2b06260b30b98fef7b99d74f3b77ca54"
server_key_version: "encryptionkey@0"
I20260621 02:47:20.778707 23315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "c2f30dd97473432ab3340fddd1d93853"
format_stamp: "Formatted at 2026-06-21 02:47:20 on dist-test-slave-hvgn"
server_key: "5505999ac55ad2a50be2425ba75efe12"
server_key_iv: "2b06260b30b98fef7b99d74f3b77ca54"
server_key_version: "encryptionkey@0"
I20260621 02:47:20.783737 23315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260621 02:47:20.786993 23330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:20.788345 23315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:47:20.788524 23315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "c2f30dd97473432ab3340fddd1d93853"
format_stamp: "Formatted at 2026-06-21 02:47:20 on dist-test-slave-hvgn"
server_key: "5505999ac55ad2a50be2425ba75efe12"
server_key_iv: "2b06260b30b98fef7b99d74f3b77ca54"
server_key_version: "encryptionkey@0"
I20260621 02:47:20.788715 23315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:20.813668 23315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:20.817718 23315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:20.818002 23315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:20.836230 23315 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:34823
I20260621 02:47:20.836314 23381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:34823 every 8 connection(s)
I20260621 02:47:20.837567 23315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:47:20.841086 23382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:47:20.846400 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 23315
I20260621 02:47:20.846616 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:47:20.847376 9880 external_mini_cluster.cc:1428] Setting key 7f2fb3b0ef70f88f21c868718d74d438
I20260621 02:47:20.847987 23382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:47:20 dist-test-slave-hvgn krb5kdc[23302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010040, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.166.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.61@KRBTEST.COM" created.
I20260621 02:47:20.874104 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:20.851829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41010 (local address 127.9.166.62:34823)
0621 02:47:20.853727 (+ 1898us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:20.853733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:20.853772 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:47:20.855995 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:20.856002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:20.856346 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:20.856578 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:20.858307 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:20.860007 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:20.862280 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:20.862815 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:20.862939 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:20.866994 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:20.867001 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:20.867013 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:20.867039 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:20.869778 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:20.870526 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:20.870534 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:20.870538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:20.870688 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:20.871057 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:20.871064 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:20.871068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:20.871611 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:20.871821 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:20.872339 (+ 518us) server_negotiation.cc:299] Negotiation successful
0621 02:47:20.872661 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1705,"spinlock_wait_cycles":73088,"thread_start_us":1478,"threads_started":1}
I20260621 02:47:20.874147 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:20.851829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34823 (local address 127.0.0.1:41010)
0621 02:47:20.852789 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:20.852840 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:20.855611 (+ 2771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:20.856614 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:20.856623 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:20.857473 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:20.857950 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:20.857976 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:20.860164 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:20.860172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:20.862115 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:20.862133 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:20.863066 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:20.863075 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:20.863294 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:20.864028 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:20.864063 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:20.866768 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:20.869922 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:20.869935 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:20.869947 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:20.870402 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:20.870795 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:20.870801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:20.870805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:20.870954 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:20.871759 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:20.871766 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:20.871976 (+ 210us) client_negotiation.cc:769] Sending connection context
0621 02:47:20.872225 (+ 249us) client_negotiation.cc:240] Negotiation successful
0621 02:47:20.872771 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":136,"thread_start_us":390,"threads_started":1}
I20260621 02:47:20.877071 23382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:47:20.878043 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:20.877668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:38133 (local address 127.0.0.1:41164)
0621 02:47:20.877832 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:20.877938 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260621 02:47:20.878262 23331 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.166.61:38133: connect: Connection refused (error 111)
I20260621 02:47:20.879833 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:20.879490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:38133 (local address 127.0.0.1:41174)
0621 02:47:20.879645 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:20.879706 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260621 02:47:20.880044 23382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.61:38133: Network error: Client connection negotiation failed: client connection to 127.9.166.61:38133: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.61.keytab.
Entry for principal kudu/127.9.166.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.61.keytab.
I20260621 02:47:20.898185 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.61:38133
--webserver_interface=127.9.166.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.61.keytab
--principal=kudu/127.9.166.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:34823,127.9.166.61:38133,127.9.166.60:40503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:47:20.929335 23382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } attempt: 1
I20260621 02:47:20.930545 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:20.930088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:38133 (local address 127.0.0.1:41184)
0621 02:47:20.930294 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:20.930415 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260621 02:47:20.931746 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:20.931331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:38133 (local address 127.0.0.1:41190)
0621 02:47:20.931522 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:20.931593 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260621 02:47:20.932011 23382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.61:38133: Network error: Client connection negotiation failed: client connection to 127.9.166.61:38133: connect: Connection refused (error 111)
I20260621 02:47:21.000286 23382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } attempt: 2
I20260621 02:47:21.001483 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.001022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:38133 (local address 127.0.0.1:41196)
0621 02:47:21.001246 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.001341 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260621 02:47:21.002532 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.002162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:38133 (local address 127.0.0.1:41204)
0621 02:47:21.002323 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.002403 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 02:47:21.002705 23382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.61:38133: Network error: Client connection negotiation failed: client connection to 127.9.166.61:38133: connect: Connection refused (error 111)
I20260621 02:47:21.093921 23382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } attempt: 3
I20260621 02:47:21.095122 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.094673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:38133 (local address 127.0.0.1:41206)
0621 02:47:21.094869 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.094975 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260621 02:47:21.096169 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.095805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:38133 (local address 127.0.0.1:41220)
0621 02:47:21.095962 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.096039 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 02:47:21.096328 23382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.61:38133: Network error: Client connection negotiation failed: client connection to 127.9.166.61:38133: connect: Connection refused (error 111)
W20260621 02:47:21.215656 23389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:21.215994 23389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:21.216164 23389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:21.223196 23389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:47:21.223335 23389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:21.223459 23389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:21.223568 23389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:47:21.223644 23389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:47:21.234414 23389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:34823,127.9.166.61:38133,127.9.166.60:40503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.61.keytab
--principal=kudu/127.9.166.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.61:38133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.166.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:21.235936 23389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:21.237242 23389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 02:47:21.239521 23382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } attempt: 4
I20260621 02:47:21.240754 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.240234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:38133 (local address 127.0.0.1:41236)
0621 02:47:21.240524 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.240621 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260621 02:47:21.241806 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.241416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:38133 (local address 127.0.0.1:41240)
0621 02:47:21.241576 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.241680 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260621 02:47:21.241979 23382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.61:38133: Network error: Client connection negotiation failed: client connection to 127.9.166.61:38133: connect: Connection refused (error 111)
W20260621 02:47:21.249107 23395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:21.249186 23394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:21.249181 23397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:21.249883 23389 server_base.cc:1034] running on GCE node
I20260621 02:47:21.250603 23389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:21.252032 23389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:21.253244 23389 hybrid_clock.cc:648] HybridClock initialized: now 1782010041253200 us; error 37 us; skew 500 ppm
I20260621 02:47:21.253607 23389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:21 dist-test-slave-hvgn krb5kdc[23302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010041, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:21.256848 23389 init.cc:377] Logged in from keytab as kudu/127.9.166.61@KRBTEST.COM (short username kudu)
I20260621 02:47:21.258395 23389 webserver.cc:460] Webserver started at http://127.9.166.61:36253/ using document root <none> and password file <none>
I20260621 02:47:21.259162 23389 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:21.259321 23389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:21.259684 23389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:47:21.262801 23389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-1/data/instance:
uuid: "e319178277d243b3a0c2780fb94932c5"
format_stamp: "Formatted at 2026-06-21 02:47:21 on dist-test-slave-hvgn"
server_key: "98d914d95a6db88bc1ce8e5d2e9d9011"
server_key_iv: "9df5bf6987c610b87763a9cd8d04e0d9"
server_key_version: "encryptionkey@0"
I20260621 02:47:21.263715 23389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-1/wal/instance:
uuid: "e319178277d243b3a0c2780fb94932c5"
format_stamp: "Formatted at 2026-06-21 02:47:21 on dist-test-slave-hvgn"
server_key: "98d914d95a6db88bc1ce8e5d2e9d9011"
server_key_iv: "9df5bf6987c610b87763a9cd8d04e0d9"
server_key_version: "encryptionkey@0"
I20260621 02:47:21.271085 23389 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260621 02:47:21.275538 23404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:21.276911 23389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:47:21.277149 23389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-1/wal
uuid: "e319178277d243b3a0c2780fb94932c5"
format_stamp: "Formatted at 2026-06-21 02:47:21 on dist-test-slave-hvgn"
server_key: "98d914d95a6db88bc1ce8e5d2e9d9011"
server_key_iv: "9df5bf6987c610b87763a9cd8d04e0d9"
server_key_version: "encryptionkey@0"
I20260621 02:47:21.277374 23389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:21.301064 23389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:21.305217 23389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:21.305456 23389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:21.325275 23389 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.61:38133
I20260621 02:47:21.325320 23455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.61:38133 every 8 connection(s)
I20260621 02:47:21.326719 23389 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-1/data/info.pb
I20260621 02:47:21.328378 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 23389
I20260621 02:47:21.328622 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-1/wal/instance
I20260621 02:47:21.329093 9880 external_mini_cluster.cc:1428] Setting key b2f33ef3704792a1ebe4a47704b7ba3b
I20260621 02:47:21.330747 23456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:47:21.339368 23456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.166.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.60@KRBTEST.COM" created.
Jun 21 02:47:21 dist-test-slave-hvgn krb5kdc[23302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010041, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.61@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:21.361384 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.342968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41018 (local address 127.9.166.62:34823)
0621 02:47:21.343240 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:21.343249 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:21.344671 (+ 1422us) server_negotiation.cc:407] Connection header received
0621 02:47:21.346689 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:21.346699 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:21.346798 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:21.346927 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:21.348530 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.349793 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.351372 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.351931 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.352035 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:21.355436 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:21.355447 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:21.355450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:21.355482 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:21.358082 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.358887 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.358891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.358893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.359017 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.359544 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.359549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.359552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.359946 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:21.360155 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:21.360972 (+ 817us) server_negotiation.cc:299] Negotiation successful
0621 02:47:21.361163 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260621 02:47:21.362805 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.343559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34823 (local address 127.0.0.1:41018)
0621 02:47:21.344448 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.344504 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:21.346371 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:21.347124 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:21.347131 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:21.347762 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:21.348335 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:21.348353 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.349950 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.349954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:21.351218 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:21.351230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.352125 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.352130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:21.352348 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:21.353088 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:21.353116 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:21.355193 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:21.358241 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:21.358250 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:21.358265 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:21.358749 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:21.359160 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:21.359166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:21.359171 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:21.359413 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:21.360132 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:21.360143 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:21.360582 (+ 439us) client_negotiation.cc:769] Sending connection context
0621 02:47:21.360883 (+ 301us) client_negotiation.cc:240] Negotiation successful
0621 02:47:21.361391 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":412,"threads_started":1}
I20260621 02:47:21.364567 23456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:47:21 dist-test-slave-hvgn krb5kdc[23302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010041, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.61@KRBTEST.COM for kudu/127.9.166.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.60.keytab.
Entry for principal kudu/127.9.166.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.60.keytab.
I20260621 02:47:21.382457 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.365281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:38133 (local address 127.0.0.1:41248)
0621 02:47:21.365496 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.365518 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:21.365639 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:21.366648 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:21.366655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:21.367142 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:21.367353 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:21.367368 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.368717 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.368725 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:21.370326 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:21.370344 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.371153 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.371160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:21.371357 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:21.371988 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:21.372016 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:21.376089 (+ 4073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:21.379638 (+ 3549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:21.379642 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:21.379645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:21.380140 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:21.380550 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:21.380554 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:21.380556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:21.380790 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:21.381446 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:21.381456 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:21.381590 (+ 134us) client_negotiation.cc:769] Sending connection context
0621 02:47:21.381782 (+ 192us) client_negotiation.cc:240] Negotiation successful
0621 02:47:21.382061 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260621 02:47:21.382447 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.60:40503
--webserver_interface=127.9.166.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.60.keytab
--principal=kudu/127.9.166.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:44865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:34823,127.9.166.61:38133,127.9.166.60:40503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:47:21.382529 23461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.365555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.9.166.61:38133)
0621 02:47:21.366203 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:21.366208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:21.366232 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:47:21.366274 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:21.366279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:21.366479 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:21.366596 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:21.367532 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.368570 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.370471 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.370976 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.371075 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:21.376297 (+ 5222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:21.376308 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:21.376321 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:21.376381 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:21.379516 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.380312 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.380318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.380320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.380430 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.380923 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.380926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.380928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.381327 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:21.381563 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:21.381912 (+ 349us) server_negotiation.cc:299] Negotiation successful
0621 02:47:21.382168 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":504,"thread_start_us":367,"threads_started":1}
I20260621 02:47:21.384919 23456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:47:21.386003 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.385575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46602)
0621 02:47:21.385752 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.385875 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260621 02:47:21.386334 23406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
I20260621 02:47:21.387876 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.387572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46614)
0621 02:47:21.387706 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.387779 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 02:47:21.388180 23456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:40503: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
I20260621 02:47:21.437511 23456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } attempt: 1
I20260621 02:47:21.438724 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.438252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46622)
0621 02:47:21.438448 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.438574 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260621 02:47:21.439796 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.439430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46632)
0621 02:47:21.439613 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.439684 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 02:47:21.439999 23456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:40503: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
I20260621 02:47:21.508219 23456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } attempt: 2
I20260621 02:47:21.509339 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.508885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46636)
0621 02:47:21.509085 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.509189 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260621 02:47:21.510288 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.509966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46652)
0621 02:47:21.510113 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.510182 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 02:47:21.510483 23456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:40503: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
I20260621 02:47:21.541253 23382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } attempt: 5
Jun 21 02:47:21 dist-test-slave-hvgn krb5kdc[23302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010040, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for kudu/127.9.166.61@KRBTEST.COM
I20260621 02:47:21.554301 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.542002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:38133 (local address 127.0.0.1:41250)
0621 02:47:21.542216 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.542237 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:21.542376 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:21.542757 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:21.542761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:21.543198 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:21.543376 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:21.543436 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.544425 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.544432 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:21.545310 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:21.545321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.546075 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.546081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:21.546266 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:21.546913 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:21.546931 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:21.548958 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:21.551976 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:21.551981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:21.551986 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:21.552330 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:21.552698 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:21.552703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:21.552707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:21.552845 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:21.553359 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:21.553364 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:21.553529 (+ 165us) client_negotiation.cc:769] Sending connection context
0621 02:47:21.553670 (+ 141us) client_negotiation.cc:240] Negotiation successful
0621 02:47:21.553940 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260621 02:47:21.554304 23461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.542102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.9.166.61:38133)
0621 02:47:21.542271 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:21.542278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:21.542300 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:47:21.542552 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:21.542557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:21.542618 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:21.542725 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:21.543607 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.544293 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.545462 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.545926 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.546062 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:21.549155 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:21.549161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:21.549164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:21.549194 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:21.551825 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.552453 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.552465 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.552468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.552588 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.552952 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.552957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.552961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.553240 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:21.553396 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:21.553779 (+ 383us) server_negotiation.cc:299] Negotiation successful
0621 02:47:21.553982 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 02:47:21.555603 23382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:47:21.556646 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.556263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46658)
0621 02:47:21.556426 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.556515 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260621 02:47:21.557760 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.557261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46674)
0621 02:47:21.557479 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.557602 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260621 02:47:21.558064 23382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:40503: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
I20260621 02:47:21.601698 23456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } attempt: 3
I20260621 02:47:21.602957 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.602487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46676)
0621 02:47:21.602708 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.602800 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260621 02:47:21.604051 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.603642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46688)
0621 02:47:21.603865 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.603939 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260621 02:47:21.604264 23456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:40503: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
I20260621 02:47:21.609295 23382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } attempt: 1
I20260621 02:47:21.610389 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.609834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46702)
0621 02:47:21.610063 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.610193 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260621 02:47:21.611557 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.611123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46714)
0621 02:47:21.611300 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.611408 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 02:47:21.611781 23382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:40503: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
I20260621 02:47:21.680030 23382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } attempt: 2
I20260621 02:47:21.681273 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.680798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46722)
0621 02:47:21.680984 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.681118 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260621 02:47:21.682353 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.682055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46724)
0621 02:47:21.682186 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.682250 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260621 02:47:21.682649 23382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:40503: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
W20260621 02:47:21.710873 23463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:21.711153 23463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:21.711326 23463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:21.718472 23463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:47:21.718569 23463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:21.718673 23463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:21.718772 23463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:47:21.718856 23463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:47:21.728797 23463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:44865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:34823,127.9.166.61:38133,127.9.166.60:40503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.60.keytab
--principal=kudu/127.9.166.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.60:40503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.166.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:21.730204 23463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:21.731549 23463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:21.743089 23468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:21.743150 23471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:21.743115 23469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:21.743959 23463 server_base.cc:1034] running on GCE node
I20260621 02:47:21.744757 23463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:21.746148 23463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:21.747344 23463 hybrid_clock.cc:648] HybridClock initialized: now 1782010041747306 us; error 33 us; skew 500 ppm
I20260621 02:47:21.747471 23456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } attempt: 4
I20260621 02:47:21.747771 23463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:47:21.748546 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.748093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46736)
0621 02:47:21.748310 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.748396 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260621 02:47:21.749637 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.749200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46744)
0621 02:47:21.749410 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.749499 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260621 02:47:21.749847 23456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:40503: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
Jun 21 02:47:21 dist-test-slave-hvgn krb5kdc[23302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010041, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:21.751206 23463 init.cc:377] Logged in from keytab as kudu/127.9.166.60@KRBTEST.COM (short username kudu)
I20260621 02:47:21.752664 23463 webserver.cc:460] Webserver started at http://127.9.166.60:46105/ using document root <none> and password file <none>
I20260621 02:47:21.753365 23463 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:21.753499 23463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:21.753805 23463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:47:21.756778 23463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-2/data/instance:
uuid: "7b0dc03669014d9797997bc492638da4"
format_stamp: "Formatted at 2026-06-21 02:47:21 on dist-test-slave-hvgn"
server_key: "c1c9da3fa60cd96e06be22bfb702dc1d"
server_key_iv: "a300966f177409c9f2ff018c5ced3a1e"
server_key_version: "encryptionkey@0"
I20260621 02:47:21.757629 23463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-2/wal/instance:
uuid: "7b0dc03669014d9797997bc492638da4"
format_stamp: "Formatted at 2026-06-21 02:47:21 on dist-test-slave-hvgn"
server_key: "c1c9da3fa60cd96e06be22bfb702dc1d"
server_key_iv: "a300966f177409c9f2ff018c5ced3a1e"
server_key_version: "encryptionkey@0"
I20260621 02:47:21.763031 23463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 02:47:21.766403 23478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:21.767963 23463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 02:47:21.768153 23463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-2/wal
uuid: "7b0dc03669014d9797997bc492638da4"
format_stamp: "Formatted at 2026-06-21 02:47:21 on dist-test-slave-hvgn"
server_key: "c1c9da3fa60cd96e06be22bfb702dc1d"
server_key_iv: "a300966f177409c9f2ff018c5ced3a1e"
server_key_version: "encryptionkey@0"
I20260621 02:47:21.768365 23463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:21.788866 23382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } attempt: 3
I20260621 02:47:21.790021 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.789524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46754)
0621 02:47:21.789767 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.789853 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260621 02:47:21.791023 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.790650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46766)
0621 02:47:21.790850 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.790914 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260621 02:47:21.791219 23382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:40503: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40503: connect: Connection refused (error 111)
I20260621 02:47:21.810393 23463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:21.814785 23463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:21.815074 23463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:21.833472 23463 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.60:40503
I20260621 02:47:21.833514 23529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.60:40503 every 8 connection(s)
I20260621 02:47:21.834832 23463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-2/data/info.pb
I20260621 02:47:21.839430 23530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:47:21.844741 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 23463
I20260621 02:47:21.844949 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACert.1782009844354615-9880-0/minicluster-data/master-2/wal/instance
I20260621 02:47:21.845402 9880 external_mini_cluster.cc:1428] Setting key ebe3f0158c26f3442c9408959d28f637
I20260621 02:47:21.848342 23530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:47:21 dist-test-slave-hvgn krb5kdc[23302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.60@KRBTEST.COM
Jun 21 02:47:21 dist-test-slave-hvgn krb5kdc[23302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010041, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.60@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:21.867422 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.851930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.9.166.62:34823)
0621 02:47:21.852178 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:21.852188 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:21.852922 (+ 734us) server_negotiation.cc:407] Connection header received
0621 02:47:21.853094 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:21.853098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:21.853158 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:21.853242 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:21.854537 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.855620 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.857564 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.858195 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.858331 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:21.862080 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:21.862085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:21.862088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:21.862120 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:21.864600 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.865300 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.865305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.865308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.865444 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.865844 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.865850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.865853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.866106 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:21.866265 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:21.866786 (+ 521us) server_negotiation.cc:299] Negotiation successful
0621 02:47:21.867026 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260621 02:47:21.868393 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.851925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:34823 (local address 127.0.0.1:41030)
0621 02:47:21.852770 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.852812 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:21.852979 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:21.853331 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:21.853344 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:21.854110 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:21.854396 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:21.854404 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.855772 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.855775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:21.857419 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:21.857431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.858422 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.858427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:21.858545 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:21.859190 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:21.859211 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:21.861882 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:21.864748 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:21.864759 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:21.864771 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:21.865173 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:21.865558 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:21.865563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:21.865566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:21.865732 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:21.866233 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:21.866243 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:21.866440 (+ 197us) client_negotiation.cc:769] Sending connection context
0621 02:47:21.866705 (+ 265us) client_negotiation.cc:240] Negotiation successful
0621 02:47:21.867203 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":464,"threads_started":1}
I20260621 02:47:21.868422 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.847215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46780 (local address 127.9.166.60:40503)
0621 02:47:21.847958 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:21.847976 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:21.848014 (+ 38us) server_negotiation.cc:407] Connection header received
0621 02:47:21.849564 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:21.849584 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:21.849983 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:21.850367 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:21.851935 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.854013 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.855883 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.856942 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.857101 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:21.861727 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:21.861737 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:21.861748 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:21.861781 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:21.864654 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.865349 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.865356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.865358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.865501 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.865887 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.865899 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.865902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.866367 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:21.866662 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:21.867002 (+ 340us) server_negotiation.cc:299] Negotiation successful
0621 02:47:21.867410 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":551,"thread_start_us":361,"threads_started":1}
I20260621 02:47:21.869951 23530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:47:21 dist-test-slave-hvgn krb5kdc[23302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010041, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.60@KRBTEST.COM for kudu/127.9.166.61@KRBTEST.COM
Jun 21 02:47:21 dist-test-slave-hvgn krb5kdc[23302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:21.884016 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.870615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:38133 (local address 127.0.0.1:41260)
0621 02:47:21.870764 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.870801 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:21.870899 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:21.871208 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:21.871213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:21.871735 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:21.871934 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:21.871948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.872947 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.872953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:21.874192 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:21.874206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.875058 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.875062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:21.875211 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:21.875775 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:21.875791 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:21.878165 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:21.881470 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:21.881475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:21.881479 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:21.881892 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:21.882338 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:21.882344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:21.882349 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:21.882546 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:21.883186 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:21.883189 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:21.883313 (+ 124us) client_negotiation.cc:769] Sending connection context
0621 02:47:21.883473 (+ 160us) client_negotiation.cc:240] Negotiation successful
0621 02:47:21.883768 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260621 02:47:21.884267 23461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.870728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41260 (local address 127.9.166.61:38133)
0621 02:47:21.870915 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:21.870921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:21.870943 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:47:21.871007 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:21.871013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:21.871083 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:21.871191 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:21.872113 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.872810 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.874340 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.874914 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.875048 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:21.878349 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:21.878361 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:21.878365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:21.878396 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:21.881345 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.882009 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.882015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.882018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.882216 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.882692 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.882698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.882701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.883063 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:21.883223 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:21.883586 (+ 363us) server_negotiation.cc:299] Negotiation successful
0621 02:47:21.883837 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:47:21.885275 23530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:47:21.888135 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.871706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.9.166.62:34823)
0621 02:47:21.871876 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:21.871880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:21.871895 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:47:21.872024 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:21.872028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:21.872121 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:21.872212 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:21.873612 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.874576 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.876242 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.876943 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.877050 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:21.881713 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:21.881717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:21.881720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:21.881748 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:21.885203 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.886000 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.886005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.886009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.886175 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.886626 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.886630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.886632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.887016 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:21.887236 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:21.887585 (+ 349us) server_negotiation.cc:299] Negotiation successful
0621 02:47:21.887823 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Jun 21 02:47:21 dist-test-slave-hvgn krb5kdc[23302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010041, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.60@KRBTEST.COM for kudu/127.9.166.60@KRBTEST.COM
Jun 21 02:47:21 dist-test-slave-hvgn krb5kdc[23302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.61@KRBTEST.COM
I20260621 02:47:21.898442 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.886149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46796)
0621 02:47:21.886418 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.886439 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:21.886589 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:21.886915 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:21.886918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:21.887335 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:21.887565 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:21.887577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.888732 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.888739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:21.889567 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:21.889577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.890455 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.890459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:21.890639 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:21.891184 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:21.891200 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:21.893332 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:21.896322 (+ 2990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:21.896328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:21.896332 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:21.896670 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:21.897049 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:21.897054 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:21.897057 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:21.897181 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:21.897644 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:21.897649 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:21.897787 (+ 138us) client_negotiation.cc:769] Sending connection context
0621 02:47:21.897939 (+ 152us) client_negotiation.cc:240] Negotiation successful
0621 02:47:21.898161 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":20}
I20260621 02:47:21.898733 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.886100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.9.166.60:40503)
0621 02:47:21.886297 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:21.886302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:21.886510 (+ 208us) server_negotiation.cc:407] Connection header received
0621 02:47:21.886717 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:21.886725 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:21.886792 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:21.886891 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:21.887699 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.888611 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.889701 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.890330 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.890459 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:21.893514 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:21.893519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:21.893522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:21.893548 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:21.896199 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.896789 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.896796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.896799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.896932 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.897280 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.897287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.897290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.897532 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:21.897680 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:21.898054 (+ 374us) server_negotiation.cc:299] Negotiation successful
0621 02:47:21.898348 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":63}
I20260621 02:47:21.904179 23530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4: Bootstrap starting.
I20260621 02:47:21.906633 23461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.889852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41270 (local address 127.9.166.61:38133)
0621 02:47:21.889997 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:21.890001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:21.890015 (+ 14us) server_negotiation.cc:407] Connection header received
0621 02:47:21.890133 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:21.890136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:21.890186 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:21.890259 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:21.891673 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.892461 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.893923 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.894606 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.894738 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:21.899059 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:21.899064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:21.899067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:21.899097 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:21.902864 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.903700 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.903708 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.903713 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.903881 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.904464 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.904472 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.904477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.904835 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:21.905140 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:21.906247 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0621 02:47:21.906410 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260621 02:47:21.907812 23530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4: Neither blocks nor log segments found. Creating new log.
I20260621 02:47:21.909101 23530 log.cc:824] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:21.911600 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:47:21.912000 23530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4: No bootstrap required, opened a new log
I20260621 02:47:21.923189 23530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } } peers { permanent_uuid: "e319178277d243b3a0c2780fb94932c5" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } } peers { permanent_uuid: "7b0dc03669014d9797997bc492638da4" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } }
I20260621 02:47:21.924644 23530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:47:21.924831 23530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b0dc03669014d9797997bc492638da4, State: Initialized, Role: FOLLOWER
I20260621 02:47:21.925791 23530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } } peers { permanent_uuid: "e319178277d243b3a0c2780fb94932c5" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } } peers { permanent_uuid: "7b0dc03669014d9797997bc492638da4" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } }
I20260621 02:47:21.930727 23543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } } peers { permanent_uuid: "e319178277d243b3a0c2780fb94932c5" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } } peers { permanent_uuid: "7b0dc03669014d9797997bc492638da4" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } } }
I20260621 02:47:21.931134 23543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:47:21.933033 23530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:47:21.943825 23461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.916927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41278 (local address 127.9.166.61:38133)
0621 02:47:21.917256 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:21.917263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:21.924066 (+ 6803us) server_negotiation.cc:407] Connection header received
0621 02:47:21.924318 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:21.924326 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:21.924432 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:21.924556 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:21.927460 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.928642 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.931090 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.931741 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.931838 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:21.935878 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:21.935885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:21.935892 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:21.935937 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:21.940333 (+ 4396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.941270 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.941275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.941279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.941424 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.941944 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.941950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.941955 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.942324 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:21.942562 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:21.943157 (+ 595us) server_negotiation.cc:299] Negotiation successful
0621 02:47:21.943428 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260621 02:47:21.946509 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.917096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46806 (local address 127.9.166.60:40503)
0621 02:47:21.917390 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:21.917396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:21.927049 (+ 9653us) server_negotiation.cc:407] Connection header received
0621 02:47:21.927238 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:21.927243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:21.927327 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:21.927465 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:21.929961 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.931074 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.933358 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.934160 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.934272 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:21.938085 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:21.938092 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:21.938097 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:21.938134 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:21.942265 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.943406 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.943411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.943415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.943568 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.944312 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.944318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.944321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.945099 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:21.945327 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:21.945895 (+ 568us) server_negotiation.cc:299] Negotiation successful
0621 02:47:21.946147 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260621 02:47:21.949426 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.916283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41044 (local address 127.9.166.62:34823)
0621 02:47:21.916470 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:21.916476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:21.929639 (+ 13163us) server_negotiation.cc:407] Connection header received
0621 02:47:21.930177 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:21.930187 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:21.930281 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:21.930359 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:21.932870 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.933936 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.936810 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.937730 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.937850 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:21.942251 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:21.942261 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:21.942268 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:21.942320 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:21.946285 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.947295 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.947303 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.947308 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.947513 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.947922 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.947927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.947931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.948205 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:21.948463 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:21.948825 (+ 362us) server_negotiation.cc:299] Negotiation successful
0621 02:47:21.949069 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260621 02:47:21.962387 23554 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:47:21.962515 23554 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:47:21.957767 23463 master_runner.cc:186] Error getting master registration for 127.9.166.62:34823: OK, instance_id { permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" instance_seqno: 1782010040831070 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 02:47:21.967043 23463 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 02:47:21.968425 23382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } attempt: 4
Jun 21 02:47:21 dist-test-slave-hvgn krb5kdc[23302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010040, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for kudu/127.9.166.60@KRBTEST.COM
I20260621 02:47:21.984941 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.968947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46820)
0621 02:47:21.969278 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:21.969297 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:21.969403 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:21.970204 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:21.970209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:21.970582 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:21.970739 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:21.970747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.972363 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.972367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:21.973359 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:21.973371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.974501 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.974505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:21.974642 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:21.975354 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:21.975376 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:21.977995 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:21.982279 (+ 4284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:21.982282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:21.982285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:21.982675 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:21.983173 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:21.983179 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:21.983183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:21.983432 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:21.984126 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:21.984131 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:21.984291 (+ 160us) client_negotiation.cc:769] Sending connection context
0621 02:47:21.984400 (+ 109us) client_negotiation.cc:240] Negotiation successful
0621 02:47:21.984623 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213}
I20260621 02:47:21.985615 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:21.969629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46820 (local address 127.9.166.60:40503)
0621 02:47:21.969829 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:21.969837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:21.969859 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:47:21.969923 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:21.969930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:21.970041 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:21.970154 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:21.970888 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.972231 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.973498 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:21.974326 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:21.974498 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:21.978189 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:21.978196 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:21.978202 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:21.978249 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:21.982152 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.982802 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.982808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.982813 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.983007 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:21.983585 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:21.983591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:21.983596 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:21.983983 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:21.984157 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:21.984932 (+ 775us) server_negotiation.cc:299] Negotiation successful
0621 02:47:21.985213 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:47:21.990078 23382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853: Bootstrap starting.
I20260621 02:47:21.993305 23382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853: Neither blocks nor log segments found. Creating new log.
I20260621 02:47:21.994376 23382 log.cc:824] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:21.998788 23382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853: No bootstrap required, opened a new log
I20260621 02:47:22.011171 23382 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } } peers { permanent_uuid: "e319178277d243b3a0c2780fb94932c5" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } } peers { permanent_uuid: "7b0dc03669014d9797997bc492638da4" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } }
I20260621 02:47:22.011737 23382 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:47:22.011888 23382 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2f30dd97473432ab3340fddd1d93853, State: Initialized, Role: FOLLOWER
I20260621 02:47:22.012486 23382 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } } peers { permanent_uuid: "e319178277d243b3a0c2780fb94932c5" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } } peers { permanent_uuid: "7b0dc03669014d9797997bc492638da4" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } }
I20260621 02:47:22.013537 23557 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } } peers { permanent_uuid: "e319178277d243b3a0c2780fb94932c5" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } } peers { permanent_uuid: "7b0dc03669014d9797997bc492638da4" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } } }
I20260621 02:47:22.013984 23557 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:47:22.014752 23382 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853 [sys.catalog]: configured and running, proceeding with master startup.
W20260621 02:47:22.026211 23568 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:47:22.026422 23568 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:47:22.025974 23315 master_runner.cc:186] Error getting master registration for 127.9.166.61:38133: OK, instance_id { permanent_uuid: "e319178277d243b3a0c2780fb94932c5" instance_seqno: 1782010041319596 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 02:47:22.038465 23315 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 02:47:22.049059 23456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } attempt: 5
Jun 21 02:47:22 dist-test-slave-hvgn krb5kdc[23302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010041, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.61@KRBTEST.COM for kudu/127.9.166.60@KRBTEST.COM
I20260621 02:47:22.064337 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:22.049624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40503 (local address 127.0.0.1:46826)
0621 02:47:22.049863 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:22.049883 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:22.050004 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:22.050340 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:22.050345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:22.050786 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:22.050945 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:22.050956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:22.052102 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:22.052110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:22.053164 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:22.053176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:22.053908 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:22.053914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:22.054134 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:22.054739 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:22.054757 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:22.056811 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:22.061856 (+ 5045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:22.061862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:22.061866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:22.062257 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:22.062643 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:22.062649 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:22.062653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:22.062796 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:22.063311 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:22.063317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:22.063500 (+ 183us) client_negotiation.cc:769] Sending connection context
0621 02:47:22.063646 (+ 146us) client_negotiation.cc:240] Negotiation successful
0621 02:47:22.063938 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260621 02:47:22.064342 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:22.049689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.9.166.60:40503)
0621 02:47:22.049866 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:22.049873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:22.049942 (+ 69us) server_negotiation.cc:407] Connection header received
0621 02:47:22.050127 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:22.050132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:22.050204 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:22.050307 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:22.051092 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:22.051968 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:22.053300 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:22.053760 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:22.053888 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:22.057010 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:22.057015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:22.057018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:22.057047 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:22.061716 (+ 4669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:22.062387 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:22.062397 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:22.062399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:22.062535 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:22.062904 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:22.062914 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:22.062917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:22.063190 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:22.063366 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:22.063738 (+ 372us) server_negotiation.cc:299] Negotiation successful
0621 02:47:22.063962 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260621 02:47:22.067205 23484 tablet.cc:2369] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:47:22.069332 23456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5: Bootstrap starting.
W20260621 02:47:22.070892 23336 tablet.cc:2369] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:47:22.073066 23456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5: Neither blocks nor log segments found. Creating new log.
I20260621 02:47:22.074150 23456 log.cc:824] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:22.083056 23456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5: No bootstrap required, opened a new log
I20260621 02:47:22.090094 23456 raft_consensus.cc:348] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } } peers { permanent_uuid: "e319178277d243b3a0c2780fb94932c5" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } } peers { permanent_uuid: "7b0dc03669014d9797997bc492638da4" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } }
I20260621 02:47:22.090430 23456 raft_consensus.cc:374] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:47:22.090581 23456 raft_consensus.cc:729] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e319178277d243b3a0c2780fb94932c5, State: Initialized, Role: FOLLOWER
I20260621 02:47:22.091187 23456 consensus_queue.cc:260] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } } peers { permanent_uuid: "e319178277d243b3a0c2780fb94932c5" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } } peers { permanent_uuid: "7b0dc03669014d9797997bc492638da4" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } }
I20260621 02:47:22.092146 23571 sys_catalog.cc:455] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } } peers { permanent_uuid: "e319178277d243b3a0c2780fb94932c5" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } } peers { permanent_uuid: "7b0dc03669014d9797997bc492638da4" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } } }
I20260621 02:47:22.092661 23571 sys_catalog.cc:458] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:47:22.093344 23456 sys_catalog.cc:564] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:47:22.115034 23389 master_runner.cc:263] No leader master found from master e319178277d243b3a0c2780fb94932c5
I20260621 02:47:22.115219 23389 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260621 02:47:22.117928 23582 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:47:22.118083 23582 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:47:22.150445 23543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:47:22.150782 23543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } } peers { permanent_uuid: "e319178277d243b3a0c2780fb94932c5" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } } peers { permanent_uuid: "7b0dc03669014d9797997bc492638da4" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } }
I20260621 02:47:22.152678 23543 leader_election.cc:288] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2f30dd97473432ab3340fddd1d93853 (127.9.166.62:34823), e319178277d243b3a0c2780fb94932c5 (127.9.166.61:38133)
I20260621 02:47:22.153779 23431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7b0dc03669014d9797997bc492638da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e319178277d243b3a0c2780fb94932c5" is_pre_election: true
I20260621 02:47:22.154394 23431 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b0dc03669014d9797997bc492638da4 in term 0.
I20260621 02:47:22.155179 23479 leader_election.cc:302] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b0dc03669014d9797997bc492638da4, e319178277d243b3a0c2780fb94932c5; no voters:
I20260621 02:47:22.156402 23357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7b0dc03669014d9797997bc492638da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2f30dd97473432ab3340fddd1d93853" is_pre_election: true
I20260621 02:47:22.156996 23357 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b0dc03669014d9797997bc492638da4 in term 0.
I20260621 02:47:22.157881 23543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:47:22.158031 23543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:47:22.158129 23543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:47:22.161775 23543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } } peers { permanent_uuid: "e319178277d243b3a0c2780fb94932c5" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } } peers { permanent_uuid: "7b0dc03669014d9797997bc492638da4" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } }
I20260621 02:47:22.162959 23357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7b0dc03669014d9797997bc492638da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2f30dd97473432ab3340fddd1d93853"
I20260621 02:47:22.163197 23357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:47:22.170382 23357 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b0dc03669014d9797997bc492638da4 in term 1.
I20260621 02:47:22.172854 23482 leader_election.cc:302] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b0dc03669014d9797997bc492638da4, c2f30dd97473432ab3340fddd1d93853; no voters:
I20260621 02:47:22.174526 23431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7b0dc03669014d9797997bc492638da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e319178277d243b3a0c2780fb94932c5"
I20260621 02:47:22.174743 23431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:47:22.178290 23431 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b0dc03669014d9797997bc492638da4 in term 1.
I20260621 02:47:22.179029 23543 leader_election.cc:288] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [CANDIDATE]: Term 1 election: Requested vote from peers c2f30dd97473432ab3340fddd1d93853 (127.9.166.62:34823), e319178277d243b3a0c2780fb94932c5 (127.9.166.61:38133)
I20260621 02:47:22.179433 23543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:47:22.179854 23543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [term 1 LEADER]: Becoming Leader. State: Replica: 7b0dc03669014d9797997bc492638da4, State: Running, Role: LEADER
I20260621 02:47:22.181062 23543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } } peers { permanent_uuid: "e319178277d243b3a0c2780fb94932c5" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } } peers { permanent_uuid: "7b0dc03669014d9797997bc492638da4" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } }
I20260621 02:47:22.196074 23584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b0dc03669014d9797997bc492638da4. Latest consensus state: current_term: 1 leader_uuid: "7b0dc03669014d9797997bc492638da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } } peers { permanent_uuid: "e319178277d243b3a0c2780fb94932c5" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } } peers { permanent_uuid: "7b0dc03669014d9797997bc492638da4" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } } }
I20260621 02:47:22.196521 23584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [sys.catalog]: This master's current role is: LEADER
I20260621 02:47:22.208101 23586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:47:22.210682 23586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:47:22.227061 23431 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5 [term 1 FOLLOWER]: Refusing update from remote peer 7b0dc03669014d9797997bc492638da4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:47:22.228191 23584 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e319178277d243b3a0c2780fb94932c5" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:47:22.235855 23357 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853 [term 1 FOLLOWER]: Refusing update from remote peer 7b0dc03669014d9797997bc492638da4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:47:22.237007 23584 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:47:22.272236 23557 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b0dc03669014d9797997bc492638da4. Latest consensus state: current_term: 1 leader_uuid: "7b0dc03669014d9797997bc492638da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } } peers { permanent_uuid: "e319178277d243b3a0c2780fb94932c5" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } } peers { permanent_uuid: "7b0dc03669014d9797997bc492638da4" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } } }
I20260621 02:47:22.272662 23557 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:47:22.277851 23571 sys_catalog.cc:455] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b0dc03669014d9797997bc492638da4. Latest consensus state: current_term: 1 leader_uuid: "7b0dc03669014d9797997bc492638da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } } peers { permanent_uuid: "e319178277d243b3a0c2780fb94932c5" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } } peers { permanent_uuid: "7b0dc03669014d9797997bc492638da4" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } } }
I20260621 02:47:22.278187 23571 sys_catalog.cc:458] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:47:22.286041 23588 mvcc.cc:205] Tried to move back new op lower bound from 7299113132948774912 to 7299113132784910336. Current Snapshot: MvccSnapshot[applied={T|T < 7299113132948774912}]
I20260621 02:47:22.288489 23589 mvcc.cc:205] Tried to move back new op lower bound from 7299113132948774912 to 7299113132784910336. Current Snapshot: MvccSnapshot[applied={T|T < 7299113132948774912}]
I20260621 02:47:22.292378 23571 sys_catalog.cc:455] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7b0dc03669014d9797997bc492638da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } } peers { permanent_uuid: "e319178277d243b3a0c2780fb94932c5" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } } peers { permanent_uuid: "7b0dc03669014d9797997bc492638da4" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } } }
I20260621 02:47:22.292645 23571 sys_catalog.cc:458] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:47:22.301568 23557 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7b0dc03669014d9797997bc492638da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } } peers { permanent_uuid: "e319178277d243b3a0c2780fb94932c5" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } } peers { permanent_uuid: "7b0dc03669014d9797997bc492638da4" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } } }
I20260621 02:47:22.301584 23584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7b0dc03669014d9797997bc492638da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } } peers { permanent_uuid: "e319178277d243b3a0c2780fb94932c5" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } } peers { permanent_uuid: "7b0dc03669014d9797997bc492638da4" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } } }
I20260621 02:47:22.301872 23584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [sys.catalog]: This master's current role is: LEADER
I20260621 02:47:22.301885 23557 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:47:22.302141 23584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7b0dc03669014d9797997bc492638da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f30dd97473432ab3340fddd1d93853" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 34823 } } peers { permanent_uuid: "e319178277d243b3a0c2780fb94932c5" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 38133 } } peers { permanent_uuid: "7b0dc03669014d9797997bc492638da4" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40503 } } }
I20260621 02:47:22.302350 23584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4 [sys.catalog]: This master's current role is: LEADER
I20260621 02:47:22.315223 23586 catalog_manager.cc:1347] Generated new cluster ID: 3c176e501f3042bd89999d13062326e8
I20260621 02:47:22.315359 23586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:47:22.371068 23586 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:47:22.374413 23586 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:47:22.421057 23586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b0dc03669014d9797997bc492638da4: Generated new TSK 0
I20260621 02:47:22.422210 23586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:47:23.028507 23568 catalog_manager.cc:1259] Loaded cluster ID: 3c176e501f3042bd89999d13062326e8
I20260621 02:47:23.028607 23568 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853: loading cluster ID for follower catalog manager: success
I20260621 02:47:23.031209 23568 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853: acquiring CA information for follower catalog manager: success
I20260621 02:47:23.033579 23568 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c2f30dd97473432ab3340fddd1d93853: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 02:47:23.120213 23582 catalog_manager.cc:1259] Loaded cluster ID: 3c176e501f3042bd89999d13062326e8
I20260621 02:47:23.120312 23582 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5: loading cluster ID for follower catalog manager: success
I20260621 02:47:23.122684 23582 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5: acquiring CA information for follower catalog manager: success
I20260621 02:47:23.124424 23582 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e319178277d243b3a0c2780fb94932c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 02:47:23.550081 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 23315
I20260621 02:47:23.551366 23377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:23.659801 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 23315
I20260621 02:47:23.682545 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 23389
I20260621 02:47:23.683698 23451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:23.790515 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 23389
I20260621 02:47:23.810989 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 23463
I20260621 02:47:23.812114 23525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:23.920100 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 23463
2026-06-21T02:47:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4856 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:47:24 dist-test-slave-hvgn krb5kdc[23605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:47:24 dist-test-slave-hvgn krb5kdc[23605](info): set up 2 sockets
Jun 21 02:47:24 dist-test-slave-hvgn krb5kdc[23605](info): commencing operation
krb5kdc: starting...
W20260621 02:47:25.015993 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:47:25 dist-test-slave-hvgn krb5kdc[23605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:47:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:47:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:47:25.234225 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:36087
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:39571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:36087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:47:25.559528 23618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:25.559894 23618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:25.560055 23618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:25.568161 23618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:47:25.568326 23618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:25.568431 23618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:25.568537 23618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:47:25.568617 23618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:47:25.579147 23618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:39571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:36087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:36087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:25.580605 23618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:25.581975 23618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:25.594259 23626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:25.594300 23623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:25.594333 23624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:25.595227 23618 server_base.cc:1034] running on GCE node
I20260621 02:47:25.596025 23618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:25.597473 23618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:25.598728 23618 hybrid_clock.cc:648] HybridClock initialized: now 1782010045598687 us; error 59 us; skew 500 ppm
I20260621 02:47:25.599095 23618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:25 dist-test-slave-hvgn krb5kdc[23605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010045, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:25.605444 23618 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:47:25.607010 23618 webserver.cc:460] Webserver started at http://127.9.166.62:42765/ using document root <none> and password file <none>
I20260621 02:47:25.607811 23618 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:25.607954 23618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:25.608263 23618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:47:25.611270 23618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "d87efaffa2f04be6ae0763d211ae2895"
format_stamp: "Formatted at 2026-06-21 02:47:25 on dist-test-slave-hvgn"
server_key: "bb91ee4a6d2b65f1d2f7e16a2521c764"
server_key_iv: "da9bcaa5e423dd0e277ae549e0b940ce"
server_key_version: "encryptionkey@0"
I20260621 02:47:25.612083 23618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "d87efaffa2f04be6ae0763d211ae2895"
format_stamp: "Formatted at 2026-06-21 02:47:25 on dist-test-slave-hvgn"
server_key: "bb91ee4a6d2b65f1d2f7e16a2521c764"
server_key_iv: "da9bcaa5e423dd0e277ae549e0b940ce"
server_key_version: "encryptionkey@0"
I20260621 02:47:25.617045 23618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:47:25.620738 23633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:25.622018 23618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:47:25.622198 23618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "d87efaffa2f04be6ae0763d211ae2895"
format_stamp: "Formatted at 2026-06-21 02:47:25 on dist-test-slave-hvgn"
server_key: "bb91ee4a6d2b65f1d2f7e16a2521c764"
server_key_iv: "da9bcaa5e423dd0e277ae549e0b940ce"
server_key_version: "encryptionkey@0"
I20260621 02:47:25.622403 23618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:25.637739 23618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:25.641954 23618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:25.642232 23618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:25.661005 23618 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:36087
I20260621 02:47:25.661034 23684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:36087 every 8 connection(s)
I20260621 02:47:25.662425 23618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:47:25.664100 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 23618
I20260621 02:47:25.664309 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:47:25.665132 9880 external_mini_cluster.cc:1428] Setting key 91bbc46047014fdbf8ddcb400f0bed4e
I20260621 02:47:25.667323 23685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:47:25 dist-test-slave-hvgn krb5kdc[23605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:25.680101 23685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895: Bootstrap starting.
I20260621 02:47:25.684258 23685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895: Neither blocks nor log segments found. Creating new log.
I20260621 02:47:25.685563 23685 log.cc:824] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:25.689167 23685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895: No bootstrap required, opened a new log
I20260621 02:47:25.689333 23688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:25.667028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39756 (local address 127.9.166.62:36087)
0621 02:47:25.667842 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:25.667854 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:25.667894 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:47:25.669580 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:25.669607 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:25.670126 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:25.670564 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:25.671914 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:25.674077 (+ 2163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:25.675415 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:25.676462 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:25.676655 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:25.680667 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:25.680676 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:25.680695 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:25.680735 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:25.684714 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:25.685407 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:25.685415 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:25.685423 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:25.685644 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:25.686056 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:25.686065 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:25.686075 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:25.686650 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:25.686946 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:25.687337 (+ 391us) server_negotiation.cc:299] Negotiation successful
0621 02:47:25.687778 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":410,"threads_started":1}
I20260621 02:47:25.693495 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:47:25.698956 23685 raft_consensus.cc:348] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87efaffa2f04be6ae0763d211ae2895" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36087 } }
I20260621 02:47:25.699337 23685 raft_consensus.cc:374] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:47:25.699487 23685 raft_consensus.cc:729] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d87efaffa2f04be6ae0763d211ae2895, State: Initialized, Role: FOLLOWER
I20260621 02:47:25.700134 23685 consensus_queue.cc:260] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87efaffa2f04be6ae0763d211ae2895" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36087 } }
I20260621 02:47:25.700387 23685 raft_consensus.cc:388] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:47:25.700523 23685 raft_consensus.cc:482] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:47:25.700728 23685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:47:25.703002 23685 raft_consensus.cc:504] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87efaffa2f04be6ae0763d211ae2895" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36087 } }
I20260621 02:47:25.703586 23685 leader_election.cc:302] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d87efaffa2f04be6ae0763d211ae2895; no voters:
I20260621 02:47:25.704319 23685 leader_election.cc:288] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:47:25.704375 23690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:47:25.705107 23690 raft_consensus.cc:686] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895 [term 1 LEADER]: Becoming Leader. State: Replica: d87efaffa2f04be6ae0763d211ae2895, State: Running, Role: LEADER
I20260621 02:47:25.705718 23690 consensus_queue.cc:237] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87efaffa2f04be6ae0763d211ae2895" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36087 } }
I20260621 02:47:25.706539 23685 sys_catalog.cc:564] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:47:25.708457 23691 sys_catalog.cc:455] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d87efaffa2f04be6ae0763d211ae2895. Latest consensus state: current_term: 1 leader_uuid: "d87efaffa2f04be6ae0763d211ae2895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87efaffa2f04be6ae0763d211ae2895" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36087 } } }
I20260621 02:47:25.708841 23691 sys_catalog.cc:458] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895 [sys.catalog]: This master's current role is: LEADER
I20260621 02:47:25.708581 23692 sys_catalog.cc:455] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d87efaffa2f04be6ae0763d211ae2895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d87efaffa2f04be6ae0763d211ae2895" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36087 } } }
I20260621 02:47:25.709108 23692 sys_catalog.cc:458] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895 [sys.catalog]: This master's current role is: LEADER
I20260621 02:47:25.711715 23695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:47:25.717133 23695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:47:25.725930 23695 catalog_manager.cc:1347] Generated new cluster ID: 8b98894e247d4e939bc55bb04cf96494
I20260621 02:47:25.726087 23695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:47:25.739204 23695 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:47:25.740556 23695 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:47:25.748059 23695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d87efaffa2f04be6ae0763d211ae2895: Generated new TSK 0
I20260621 02:47:25.749091 23695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:47:25.805298 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 23618
I20260621 02:47:25.806460 23680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:25.903162 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 23618
2026-06-21T02:47:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1978 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:47:25 dist-test-slave-hvgn krb5kdc[23713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:47:25 dist-test-slave-hvgn krb5kdc[23713](info): set up 2 sockets
Jun 21 02:47:25 dist-test-slave-hvgn krb5kdc[23713](info): commencing operation
krb5kdc: starting...
W20260621 02:47:27.103940 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:47:27 dist-test-slave-hvgn krb5kdc[23713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:47:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:47:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:47:27.341143 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:38157
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:38157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:47:27.666004 23726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:27.666293 23726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:27.666486 23726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:27.673810 23726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:47:27.673923 23726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:27.674022 23726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:27.674140 23726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:47:27.674211 23726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:47:27.684155 23726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:38157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:38157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:27.685503 23726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:27.686878 23726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:27.698594 23731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:27.698644 23732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:27.698661 23734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:27.699679 23726 server_base.cc:1034] running on GCE node
I20260621 02:47:27.700517 23726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:27.701931 23726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:27.703174 23726 hybrid_clock.cc:648] HybridClock initialized: now 1782010047703116 us; error 51 us; skew 500 ppm
I20260621 02:47:27.703611 23726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:27 dist-test-slave-hvgn krb5kdc[23713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010047, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:27.707106 23726 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:47:27.708876 23726 webserver.cc:460] Webserver started at http://127.9.166.62:35895/ using document root <none> and password file <none>
I20260621 02:47:27.709599 23726 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:27.709748 23726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:27.710059 23726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:47:27.713182 23726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "7a60191689bc4a61ac3a97d4d5091569"
format_stamp: "Formatted at 2026-06-21 02:47:27 on dist-test-slave-hvgn"
server_key: "4b364f48819db996df814204054ff284"
server_key_iv: "47e72cd5cabbd3c02dbc3fcfe9b01f82"
server_key_version: "encryptionkey@0"
I20260621 02:47:27.713954 23726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "7a60191689bc4a61ac3a97d4d5091569"
format_stamp: "Formatted at 2026-06-21 02:47:27 on dist-test-slave-hvgn"
server_key: "4b364f48819db996df814204054ff284"
server_key_iv: "47e72cd5cabbd3c02dbc3fcfe9b01f82"
server_key_version: "encryptionkey@0"
I20260621 02:47:27.719234 23726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260621 02:47:27.722666 23741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:27.724021 23726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:47:27.724200 23726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "7a60191689bc4a61ac3a97d4d5091569"
format_stamp: "Formatted at 2026-06-21 02:47:27 on dist-test-slave-hvgn"
server_key: "4b364f48819db996df814204054ff284"
server_key_iv: "47e72cd5cabbd3c02dbc3fcfe9b01f82"
server_key_version: "encryptionkey@0"
I20260621 02:47:27.724390 23726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:27.736435 23726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:27.740566 23726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:27.740849 23726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:27.759562 23726 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:38157
I20260621 02:47:27.759653 23792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:38157 every 8 connection(s)
I20260621 02:47:27.760957 23726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:47:27.763180 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 23726
I20260621 02:47:27.763433 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:47:27.764297 9880 external_mini_cluster.cc:1428] Setting key 611c6562abb793bcf5ab682e2f65d8ae
I20260621 02:47:27.765319 23793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:47:27.777613 23793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569: Bootstrap starting.
Jun 21 02:47:27 dist-test-slave-hvgn krb5kdc[23713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:27.781322 23793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569: Neither blocks nor log segments found. Creating new log.
I20260621 02:47:27.782407 23793 log.cc:824] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:27.785828 23793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569: No bootstrap required, opened a new log
I20260621 02:47:27.786345 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:27.766470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50404 (local address 127.9.166.62:38157)
0621 02:47:27.767318 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:27.767329 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:27.767369 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:47:27.768953 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:27.768982 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:27.769363 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:27.769774 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:27.771166 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:27.772555 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:27.773822 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:27.774620 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:27.774815 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:27.779181 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:27.779188 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:27.779202 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:27.779243 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:27.782267 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:27.783144 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:27.783152 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:27.783160 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:27.783370 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:27.783764 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:27.783770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:27.783773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:27.784278 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:27.784498 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:27.784931 (+ 433us) server_negotiation.cc:299] Negotiation successful
0621 02:47:27.785294 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":432,"threads_started":1}
I20260621 02:47:27.793399 23793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60191689bc4a61ac3a97d4d5091569" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38157 } }
I20260621 02:47:27.793776 23793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:47:27.793896 23793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a60191689bc4a61ac3a97d4d5091569, State: Initialized, Role: FOLLOWER
I20260621 02:47:27.794445 23793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60191689bc4a61ac3a97d4d5091569" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38157 } }
I20260621 02:47:27.794684 23793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:47:27.794828 23793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:47:27.794992 23793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:47:27.797281 23793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60191689bc4a61ac3a97d4d5091569" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38157 } }
I20260621 02:47:27.797811 23793 leader_election.cc:302] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a60191689bc4a61ac3a97d4d5091569; no voters:
I20260621 02:47:27.798408 23793 leader_election.cc:288] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:47:27.798583 23798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:47:27.799253 23798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 1 LEADER]: Becoming Leader. State: Replica: 7a60191689bc4a61ac3a97d4d5091569, State: Running, Role: LEADER
I20260621 02:47:27.799834 23798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60191689bc4a61ac3a97d4d5091569" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38157 } }
I20260621 02:47:27.800560 23793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:47:27.802546 23800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a60191689bc4a61ac3a97d4d5091569. Latest consensus state: current_term: 1 leader_uuid: "7a60191689bc4a61ac3a97d4d5091569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60191689bc4a61ac3a97d4d5091569" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38157 } } }
I20260621 02:47:27.802929 23800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [sys.catalog]: This master's current role is: LEADER
I20260621 02:47:27.802673 23799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a60191689bc4a61ac3a97d4d5091569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60191689bc4a61ac3a97d4d5091569" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38157 } } }
I20260621 02:47:27.803407 23799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [sys.catalog]: This master's current role is: LEADER
I20260621 02:47:27.804067 23805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:47:27.812888 23805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:47:27.823288 23805 catalog_manager.cc:1347] Generated new cluster ID: 2e6276e78ee74f188bdcc84b93d976ed
I20260621 02:47:27.823469 23805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:47:27.836830 23805 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:47:27.838137 23805 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:47:27.845652 23805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569: Generated new TSK 0
I20260621 02:47:27.846541 23805 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
I20260621 02:47:27.903642 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:38157
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:47:28.225898 23819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:28.226162 23819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:28.226349 23819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:28.233613 23819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:28.233757 23819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:28.233945 23819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:47:28.243319 23819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:28.244648 23819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:28.245942 23819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:28.258420 23824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:28.258452 23827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:28.258526 23825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:28.259794 23819 server_base.cc:1034] running on GCE node
I20260621 02:47:28.260555 23819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:28.261726 23819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:28.262996 23819 hybrid_clock.cc:648] HybridClock initialized: now 1782010048262949 us; error 62 us; skew 500 ppm
I20260621 02:47:28.263324 23819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:28 dist-test-slave-hvgn krb5kdc[23713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010048, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:28.266475 23819 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:47:28.268011 23819 webserver.cc:460] Webserver started at http://127.9.166.1:33165/ using document root <none> and password file <none>
I20260621 02:47:28.268704 23819 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:28.268851 23819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:28.269163 23819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:47:28.272045 23819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "a68b070367da4cbcb3865225c1c034be"
format_stamp: "Formatted at 2026-06-21 02:47:28 on dist-test-slave-hvgn"
server_key: "3eec10c6e6d3f9330ab479dbb51e277e"
server_key_iv: "4289d9c704cfb78f61182b693e23a485"
server_key_version: "encryptionkey@0"
I20260621 02:47:28.272802 23819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "a68b070367da4cbcb3865225c1c034be"
format_stamp: "Formatted at 2026-06-21 02:47:28 on dist-test-slave-hvgn"
server_key: "3eec10c6e6d3f9330ab479dbb51e277e"
server_key_iv: "4289d9c704cfb78f61182b693e23a485"
server_key_version: "encryptionkey@0"
I20260621 02:47:28.277810 23819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:47:28.281270 23834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:28.282527 23819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260621 02:47:28.282704 23819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "a68b070367da4cbcb3865225c1c034be"
format_stamp: "Formatted at 2026-06-21 02:47:28 on dist-test-slave-hvgn"
server_key: "3eec10c6e6d3f9330ab479dbb51e277e"
server_key_iv: "4289d9c704cfb78f61182b693e23a485"
server_key_version: "encryptionkey@0"
I20260621 02:47:28.282903 23819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:28.310205 23819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:28.314098 23819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:28.314384 23819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:28.315313 23819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:47:28.316758 23819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:47:28.316860 23819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:28.316984 23819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:47:28.317039 23819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:28.350356 23819 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:43803
I20260621 02:47:28.350390 23946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:43803 every 8 connection(s)
I20260621 02:47:28.351682 23819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:47:28.355909 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 23819
I20260621 02:47:28.356127 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:47:28.357146 9880 external_mini_cluster.cc:1428] Setting key 14c63aecccf9d319209e53f19f340d54
Jun 21 02:47:28 dist-test-slave-hvgn krb5kdc[23713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010048, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:28.375629 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:28.354191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:35133 (local address 127.9.166.62:38157)
0621 02:47:28.354632 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:28.354637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:28.355568 (+ 931us) server_negotiation.cc:407] Connection header received
0621 02:47:28.358362 (+ 2794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:28.358367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:28.358440 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:28.358557 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:28.360531 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:28.361228 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:28.363744 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:28.364140 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:28.364239 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:28.368555 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:28.368568 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:28.368576 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:28.368624 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:28.372238 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:28.372991 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:28.372999 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:28.373006 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:28.373153 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:28.373638 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:28.373646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:28.373652 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:28.374032 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:28.374200 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:28.374876 (+ 676us) server_negotiation.cc:299] Negotiation successful
0621 02:47:28.375122 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":262,"threads_started":1}
I20260621 02:47:28.376448 23950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:28.354580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.1:35133)
0621 02:47:28.355328 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:28.355372 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:28.358048 (+ 2676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:28.358770 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:28.358777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:28.359526 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:28.360344 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:28.360383 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:28.361397 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:28.361405 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:28.363574 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:28.363592 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:28.364389 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:28.364400 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:28.364750 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:28.365494 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:28.365528 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:28.368311 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:28.372411 (+ 4100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:28.372418 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:28.372431 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:28.372851 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:28.373311 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:28.373315 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:28.373317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:28.373509 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:28.374159 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:28.374165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:28.374487 (+ 322us) client_negotiation.cc:769] Sending connection context
0621 02:47:28.374708 (+ 221us) client_negotiation.cc:240] Negotiation successful
0621 02:47:28.375126 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":312,"threads_started":1}
I20260621 02:47:28.378005 23947 heartbeater.cc:347] Connected to a master server at 127.9.166.62:38157
I20260621 02:47:28.378326 23947 heartbeater.cc:464] Registering TS with master...
I20260621 02:47:28.379215 23947 heartbeater.cc:511] Master 127.9.166.62:38157 requested a full tablet report, sending...
I20260621 02:47:28.381557 23758 ts_manager.cc:194] Registered new tserver with Master: a68b070367da4cbcb3865225c1c034be (127.9.166.1:43803)
WARNING: no policy specified for kudu/127.9.166.2@KRBTEST.COM; defaulting to no policy
I20260621 02:47:28.383140 23758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:35133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
I20260621 02:47:28.410154 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:38157
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:47:28.737100 23953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:28.737368 23953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:28.737524 23953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:28.744884 23953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:28.745007 23953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:28.745210 23953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:47:28.755007 23953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:28.756410 23953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:28.757723 23953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:28.772691 23959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:28.773217 23958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:28.772668 23961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:28.773847 23953 server_base.cc:1034] running on GCE node
I20260621 02:47:28.774438 23953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:28.775434 23953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:28.776657 23953 hybrid_clock.cc:648] HybridClock initialized: now 1782010048776594 us; error 54 us; skew 500 ppm
I20260621 02:47:28.777081 23953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:28 dist-test-slave-hvgn krb5kdc[23713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010048, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:28.780215 23953 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:47:28.781648 23953 webserver.cc:460] Webserver started at http://127.9.166.2:44679/ using document root <none> and password file <none>
I20260621 02:47:28.782267 23953 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:28.782362 23953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:28.782608 23953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:47:28.785421 23953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "3c30f49ccb1940908d297bbca090b0d5"
format_stamp: "Formatted at 2026-06-21 02:47:28 on dist-test-slave-hvgn"
server_key: "aaebd01a5b7a68ef504e264e2b8aaaef"
server_key_iv: "893956cb596c04d73e60a6d2664b2d7c"
server_key_version: "encryptionkey@0"
I20260621 02:47:28.786094 23953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "3c30f49ccb1940908d297bbca090b0d5"
format_stamp: "Formatted at 2026-06-21 02:47:28 on dist-test-slave-hvgn"
server_key: "aaebd01a5b7a68ef504e264e2b8aaaef"
server_key_iv: "893956cb596c04d73e60a6d2664b2d7c"
server_key_version: "encryptionkey@0"
I20260621 02:47:28.791102 23953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260621 02:47:28.794325 23968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:28.795612 23953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 02:47:28.795765 23953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "3c30f49ccb1940908d297bbca090b0d5"
format_stamp: "Formatted at 2026-06-21 02:47:28 on dist-test-slave-hvgn"
server_key: "aaebd01a5b7a68ef504e264e2b8aaaef"
server_key_iv: "893956cb596c04d73e60a6d2664b2d7c"
server_key_version: "encryptionkey@0"
I20260621 02:47:28.795931 23953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:28.813664 23953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:28.817999 23953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:28.818295 23953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:28.819181 23953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:47:28.820535 23953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:47:28.820606 23953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:28.820752 23953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:47:28.820808 23953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:28.854271 23953 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:36195
I20260621 02:47:28.854319 24080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:36195 every 8 connection(s)
I20260621 02:47:28.855562 23953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:47:28.861181 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 23953
I20260621 02:47:28.861377 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:47:28.861915 9880 external_mini_cluster.cc:1428] Setting key 80c1fa30715042c57a640c6401a080c5
Jun 21 02:47:28 dist-test-slave-hvgn krb5kdc[23713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010048, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:28.875735 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:28.858268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:46101 (local address 127.9.166.62:38157)
0621 02:47:28.858491 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:28.858497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:28.859681 (+ 1184us) server_negotiation.cc:407] Connection header received
0621 02:47:28.861434 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:28.861444 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:28.861547 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:28.861661 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:28.863273 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:28.864123 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:28.865637 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:28.866078 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:28.866218 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:28.869884 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:28.869890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:28.869894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:28.869927 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:28.872584 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:28.873349 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:28.873355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:28.873358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:28.873504 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:28.873952 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:28.873957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:28.873961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:28.874226 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:28.874385 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:28.875081 (+ 696us) server_negotiation.cc:299] Negotiation successful
0621 02:47:28.875310 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":83}
I20260621 02:47:28.876807 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:28.858748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.2:46101)
0621 02:47:28.859514 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:28.859557 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:28.861150 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:28.861857 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:28.861864 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:28.862421 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:28.863098 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:28.863117 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:28.864282 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:28.864288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:28.865487 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:28.865499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:28.866314 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:28.866321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:28.866557 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:28.867245 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:28.867271 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:28.869658 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:28.872745 (+ 3087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:28.872753 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:28.872769 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:28.873213 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:28.873622 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:28.873626 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:28.873630 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:28.873815 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:28.874358 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:28.874366 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:28.874709 (+ 343us) client_negotiation.cc:769] Sending connection context
0621 02:47:28.875000 (+ 291us) client_negotiation.cc:240] Negotiation successful
0621 02:47:28.875550 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":261,"threads_started":1}
I20260621 02:47:28.878296 24081 heartbeater.cc:347] Connected to a master server at 127.9.166.62:38157
I20260621 02:47:28.878609 24081 heartbeater.cc:464] Registering TS with master...
I20260621 02:47:28.879319 24081 heartbeater.cc:511] Master 127.9.166.62:38157 requested a full tablet report, sending...
I20260621 02:47:28.880771 23758 ts_manager.cc:194] Registered new tserver with Master: 3c30f49ccb1940908d297bbca090b0d5 (127.9.166.2:36195)
I20260621 02:47:28.881590 23758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:46101
WARNING: no policy specified for kudu/127.9.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
I20260621 02:47:28.912583 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:38157
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:47:29.232034 24086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:29.232300 24086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:29.232393 24086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:29.239439 24086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:29.239547 24086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:29.239655 24086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:47:29.249192 24086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:29.250396 24086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:29.251626 24086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:29.265226 24091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:29.265281 24092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:29.265800 24094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:29.270212 24086 server_base.cc:1034] running on GCE node
I20260621 02:47:29.270951 24086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:29.272236 24086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:29.273473 24086 hybrid_clock.cc:648] HybridClock initialized: now 1782010049273409 us; error 53 us; skew 500 ppm
I20260621 02:47:29.273840 24086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:29 dist-test-slave-hvgn krb5kdc[23713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010049, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:29.277168 24086 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:47:29.278594 24086 webserver.cc:460] Webserver started at http://127.9.166.3:37147/ using document root <none> and password file <none>
I20260621 02:47:29.279211 24086 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:29.279315 24086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:29.279631 24086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:47:29.282410 24086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "77a4a103a8d645a6a75308c18cbb6e94"
format_stamp: "Formatted at 2026-06-21 02:47:29 on dist-test-slave-hvgn"
server_key: "aa4f49132c6e75f7c89f6753614eeb77"
server_key_iv: "105b0a06cb7cabca667668775b5227e7"
server_key_version: "encryptionkey@0"
I20260621 02:47:29.283084 24086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "77a4a103a8d645a6a75308c18cbb6e94"
format_stamp: "Formatted at 2026-06-21 02:47:29 on dist-test-slave-hvgn"
server_key: "aa4f49132c6e75f7c89f6753614eeb77"
server_key_iv: "105b0a06cb7cabca667668775b5227e7"
server_key_version: "encryptionkey@0"
I20260621 02:47:29.288405 24086 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:47:29.291780 24101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:29.293061 24086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:47:29.293188 24086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "77a4a103a8d645a6a75308c18cbb6e94"
format_stamp: "Formatted at 2026-06-21 02:47:29 on dist-test-slave-hvgn"
server_key: "aa4f49132c6e75f7c89f6753614eeb77"
server_key_iv: "105b0a06cb7cabca667668775b5227e7"
server_key_version: "encryptionkey@0"
I20260621 02:47:29.293346 24086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:29.321760 24086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:29.325640 24086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:29.325871 24086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:29.326730 24086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:47:29.328190 24086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:47:29.328260 24086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:29.328346 24086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:47:29.328399 24086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:29.361438 24086 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:34067
I20260621 02:47:29.361483 24213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:34067 every 8 connection(s)
I20260621 02:47:29.362775 24086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:47:29.364809 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 24086
I20260621 02:47:29.365023 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:47:29.365585 9880 external_mini_cluster.cc:1428] Setting key 8065633906445fdde2b54d794b64c15d
Jun 21 02:47:29 dist-test-slave-hvgn krb5kdc[23713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010049, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:29.383165 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:29.365527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:47773 (local address 127.9.166.62:38157)
0621 02:47:29.365745 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:29.365752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:29.367171 (+ 1419us) server_negotiation.cc:407] Connection header received
0621 02:47:29.369062 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:29.369065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:29.369154 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:29.369235 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:29.370710 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.371529 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.373184 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.373591 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.373686 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:29.377333 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:29.377338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:29.377341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:29.377382 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:29.380409 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:29.381200 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:29.381205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:29.381209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:29.381343 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:29.381724 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:29.381727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:29.381729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:29.381990 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:29.382111 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:29.382773 (+ 662us) server_negotiation.cc:299] Negotiation successful
0621 02:47:29.382929 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 02:47:29.384195 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:29.366177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.3:47773)
0621 02:47:29.367014 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:29.367059 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:29.368802 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:29.369447 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:29.369455 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:29.370005 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:29.370567 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:29.370586 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.371688 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.371693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:29.373052 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:29.373064 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.373822 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.373827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:29.374041 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:29.374779 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:29.374814 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:29.377112 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:29.380563 (+ 3451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:29.380574 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:29.380589 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:29.381093 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:29.381457 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:29.381460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:29.381462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:29.381632 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:29.382104 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:29.382110 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:29.382465 (+ 355us) client_negotiation.cc:769] Sending connection context
0621 02:47:29.382688 (+ 223us) client_negotiation.cc:240] Negotiation successful
0621 02:47:29.383053 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":351,"threads_started":1}
I20260621 02:47:29.385579 24214 heartbeater.cc:347] Connected to a master server at 127.9.166.62:38157
I20260621 02:47:29.385921 24214 heartbeater.cc:464] Registering TS with master...
I20260621 02:47:29.386291 23947 heartbeater.cc:503] Master 127.9.166.62:38157 was elected leader, sending a full tablet report...
I20260621 02:47:29.386731 24214 heartbeater.cc:511] Master 127.9.166.62:38157 requested a full tablet report, sending...
I20260621 02:47:29.388294 23758 ts_manager.cc:194] Registered new tserver with Master: 77a4a103a8d645a6a75308c18cbb6e94 (127.9.166.3:34067)
I20260621 02:47:29.389043 23758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:47773
I20260621 02:47:29.393319 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:47:29.410481 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:29.397900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50420 (local address 127.9.166.62:38157)
0621 02:47:29.398091 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:29.398095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:29.398572 (+ 477us) server_negotiation.cc:407] Connection header received
0621 02:47:29.398696 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:29.398699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:29.398772 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:29.398863 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:29.400429 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.401256 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.402723 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.403121 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.403216 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:29.405582 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:29.405587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:29.405591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:29.405623 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:29.408090 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:29.408775 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:29.408779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:29.408782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:29.408908 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:29.409253 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:29.409256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:29.409258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:29.409558 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:29.409764 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:29.410085 (+ 321us) server_negotiation.cc:299] Negotiation successful
0621 02:47:29.410263 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 02:47:29.414597 23758 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:47:29.421833 23758 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:47:29.445248 24230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:29.434939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:43803 (local address 127.0.0.1:34058)
0621 02:47:29.437669 (+ 2730us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:29.437683 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:29.437846 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:29.438762 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:29.438769 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:29.438799 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:29.439213 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:29.439224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.440833 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.440841 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:29.442646 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:29.442659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.444169 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.444175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:29.444362 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:29.444423 (+ 61us) client_negotiation.cc:769] Sending connection context
0621 02:47:29.444564 (+ 141us) client_negotiation.cc:240] Negotiation successful
0621 02:47:29.444770 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2603,"mutex_wait_us":9,"thread_start_us":2470,"threads_started":1}
I20260621 02:47:29.445644 24226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:29.433561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:34067 (local address 127.0.0.1:51734)
0621 02:47:29.434353 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:29.434401 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:29.434569 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:29.437552 (+ 2983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:29.437559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:29.437587 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:29.438110 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:29.438124 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.439990 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.439998 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:29.442686 (+ 2688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:29.442700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.444638 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.444645 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:29.444905 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:29.444937 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 02:47:29.445018 (+ 81us) client_negotiation.cc:240] Negotiation successful
0621 02:47:29.445160 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":172,"thread_start_us":366,"threads_started":1}
I20260621 02:47:29.445691 24228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:29.433748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51734 (local address 127.9.166.3:34067)
0621 02:47:29.436979 (+ 3231us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:29.436990 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:29.437023 (+ 33us) server_negotiation.cc:407] Connection header received
0621 02:47:29.437119 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:29.437127 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:29.437361 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:29.438116 (+ 755us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:29.438384 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.439864 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.442925 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.444370 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.444898 (+ 528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:29.445124 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:29.445189 (+ 65us) server_negotiation.cc:299] Negotiation successful
0621 02:47:29.445365 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3057,"thread_start_us":2885,"threads_started":1}
I20260621 02:47:29.445993 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:29.435447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.9.166.1:43803)
0621 02:47:29.437302 (+ 1855us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:29.437309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:29.437795 (+ 486us) server_negotiation.cc:407] Connection header received
0621 02:47:29.438049 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:29.438059 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:29.438281 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:29.438494 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:29.439425 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.440687 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.442858 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.443968 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.444116 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:29.444363 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:29.445495 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0621 02:47:29.445677 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1709,"thread_start_us":1562,"threads_started":1}
I20260621 02:47:29.446099 24229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:29.434249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52586 (local address 127.9.166.2:36195)
0621 02:47:29.435796 (+ 1547us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:29.435807 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:29.435843 (+ 36us) server_negotiation.cc:407] Connection header received
0621 02:47:29.435961 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:29.435972 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:29.436275 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:29.437856 (+ 1581us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:29.438431 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.440085 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.442935 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.444366 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.444503 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:29.444711 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:29.445296 (+ 585us) server_negotiation.cc:299] Negotiation successful
0621 02:47:29.445773 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1374,"thread_start_us":1276,"threads_started":1}
I20260621 02:47:29.447263 24227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:29.433561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:36195 (local address 127.0.0.1:52586)
0621 02:47:29.435155 (+ 1594us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:29.435175 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:29.435283 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:29.437460 (+ 2177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:29.437470 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:29.437509 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:29.438182 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:29.438194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.440214 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.440219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:29.442760 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:29.442775 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.444612 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.444619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:29.444790 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:29.444867 (+ 77us) client_negotiation.cc:769] Sending connection context
0621 02:47:29.444962 (+ 95us) client_negotiation.cc:240] Negotiation successful
0621 02:47:29.445095 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1444,"mutex_wait_us":555,"thread_start_us":1074,"threads_started":1}
I20260621 02:47:29.449898 23882 tablet_service.cc:1467] Processing CreateTablet for tablet 86140a1bc7f64812b75dd0a490750077 (DEFAULT_TABLE table=test-table [id=a1bc5d98180d47be90ef7c905bf81969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:47:29.449898 24149 tablet_service.cc:1467] Processing CreateTablet for tablet 86140a1bc7f64812b75dd0a490750077 (DEFAULT_TABLE table=test-table [id=a1bc5d98180d47be90ef7c905bf81969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:47:29.451403 23882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86140a1bc7f64812b75dd0a490750077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:47:29.451423 24149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86140a1bc7f64812b75dd0a490750077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:47:29.453562 24016 tablet_service.cc:1467] Processing CreateTablet for tablet 86140a1bc7f64812b75dd0a490750077 (DEFAULT_TABLE table=test-table [id=a1bc5d98180d47be90ef7c905bf81969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:47:29.454840 24016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86140a1bc7f64812b75dd0a490750077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:47:29.465953 24232 tablet_bootstrap.cc:492] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5: Bootstrap starting.
I20260621 02:47:29.467904 24233 tablet_bootstrap.cc:492] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be: Bootstrap starting.
I20260621 02:47:29.469626 24234 tablet_bootstrap.cc:492] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94: Bootstrap starting.
I20260621 02:47:29.471160 24232 tablet_bootstrap.cc:654] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5: Neither blocks nor log segments found. Creating new log.
I20260621 02:47:29.471431 24233 tablet_bootstrap.cc:654] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be: Neither blocks nor log segments found. Creating new log.
I20260621 02:47:29.472481 24234 tablet_bootstrap.cc:654] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94: Neither blocks nor log segments found. Creating new log.
I20260621 02:47:29.472524 24233 log.cc:824] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:29.472524 24232 log.cc:824] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:29.473657 24234 log.cc:824] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:29.475652 24233 tablet_bootstrap.cc:492] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be: No bootstrap required, opened a new log
I20260621 02:47:29.475652 24232 tablet_bootstrap.cc:492] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5: No bootstrap required, opened a new log
I20260621 02:47:29.475965 24233 ts_tablet_manager.cc:1397] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260621 02:47:29.475965 24232 ts_tablet_manager.cc:1397] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260621 02:47:29.476634 24234 tablet_bootstrap.cc:492] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94: No bootstrap required, opened a new log
I20260621 02:47:29.476881 24234 ts_tablet_manager.cc:1397] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260621 02:47:29.483299 24233 raft_consensus.cc:348] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 } } peers { permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 } } peers { permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 } }
I20260621 02:47:29.483723 24233 raft_consensus.cc:374] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:47:29.483829 24233 raft_consensus.cc:729] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68b070367da4cbcb3865225c1c034be, State: Initialized, Role: FOLLOWER
I20260621 02:47:29.484445 24233 consensus_queue.cc:260] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 } } peers { permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 } } peers { permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 } }
I20260621 02:47:29.486234 24233 ts_tablet_manager.cc:1428] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260621 02:47:29.486680 24232 raft_consensus.cc:348] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 } } peers { permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 } } peers { permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 } }
I20260621 02:47:29.487099 24232 raft_consensus.cc:374] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:47:29.487198 24232 raft_consensus.cc:729] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c30f49ccb1940908d297bbca090b0d5, State: Initialized, Role: FOLLOWER
I20260621 02:47:29.487615 24234 raft_consensus.cc:348] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 } } peers { permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 } } peers { permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 } }
I20260621 02:47:29.488054 24234 raft_consensus.cc:374] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:47:29.488144 24234 raft_consensus.cc:729] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77a4a103a8d645a6a75308c18cbb6e94, State: Initialized, Role: FOLLOWER
I20260621 02:47:29.487962 24232 consensus_queue.cc:260] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 } } peers { permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 } } peers { permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 } }
I20260621 02:47:29.488778 24234 consensus_queue.cc:260] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 } } peers { permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 } } peers { permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 } }
I20260621 02:47:29.492075 24081 heartbeater.cc:503] Master 127.9.166.62:38157 was elected leader, sending a full tablet report...
I20260621 02:47:29.492964 24232 ts_tablet_manager.cc:1428] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260621 02:47:29.493119 24234 ts_tablet_manager.cc:1428] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260621 02:47:29.493240 24214 heartbeater.cc:503] Master 127.9.166.62:38157 was elected leader, sending a full tablet report...
I20260621 02:47:29.573510 24239 raft_consensus.cc:482] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:47:29.573772 24239 raft_consensus.cc:504] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 } } peers { permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 } } peers { permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 } }
I20260621 02:47:29.575347 24239 leader_election.cc:288] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77a4a103a8d645a6a75308c18cbb6e94 (127.9.166.3:34067), a68b070367da4cbcb3865225c1c034be (127.9.166.1:43803)
I20260621 02:47:29.582580 24228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:29.575878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:41207 (local address 127.9.166.3:34067)
0621 02:47:29.576130 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:29.576138 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:29.576476 (+ 338us) server_negotiation.cc:407] Connection header received
0621 02:47:29.576733 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:29.576741 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:29.576845 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:29.576948 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:29.577641 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.578809 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.580680 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.581563 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.581660 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:29.581743 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:29.582173 (+ 430us) server_negotiation.cc:299] Negotiation successful
0621 02:47:29.582308 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:47:29.582602 24241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:29.575678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:34067 (local address 127.9.166.2:41207)
0621 02:47:29.576364 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:29.576386 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:29.576584 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:29.577012 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:29.577018 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:29.577053 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:29.577470 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:29.577482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.578939 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.578945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:29.580529 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:29.580542 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.581785 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.581791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:29.581973 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:29.582012 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 02:47:29.582109 (+ 97us) client_negotiation.cc:240] Negotiation successful
0621 02:47:29.582218 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":1,"thread_start_us":302,"threads_started":1}
I20260621 02:47:29.582896 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:29.576733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:47129 (local address 127.9.166.1:43803)
0621 02:47:29.577004 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:29.577010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:29.577197 (+ 187us) server_negotiation.cc:407] Connection header received
0621 02:47:29.577434 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:29.577440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:29.577518 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:29.577608 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:29.578257 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.579008 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.580930 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.581853 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.581983 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:29.582077 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:29.582496 (+ 419us) server_negotiation.cc:299] Negotiation successful
0621 02:47:29.582643 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260621 02:47:29.583004 24242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:29.576403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:43803 (local address 127.9.166.2:47129)
0621 02:47:29.577126 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:29.577139 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:29.577292 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:29.577634 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:29.577638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:29.577661 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:29.578042 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:29.578054 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.579162 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.579166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:29.580772 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:29.580787 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.582062 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.582067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:29.582308 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:29.582345 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 02:47:29.582442 (+ 97us) client_negotiation.cc:240] Negotiation successful
0621 02:47:29.582607 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":21,"spinlock_wait_cycles":17152,"thread_start_us":618,"threads_started":1}
I20260621 02:47:29.583870 24169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86140a1bc7f64812b75dd0a490750077" candidate_uuid: "3c30f49ccb1940908d297bbca090b0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77a4a103a8d645a6a75308c18cbb6e94" is_pre_election: true
I20260621 02:47:29.583930 23902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86140a1bc7f64812b75dd0a490750077" candidate_uuid: "3c30f49ccb1940908d297bbca090b0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68b070367da4cbcb3865225c1c034be" is_pre_election: true
I20260621 02:47:29.584369 24169 raft_consensus.cc:2454] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c30f49ccb1940908d297bbca090b0d5 in term 0.
I20260621 02:47:29.584398 23902 raft_consensus.cc:2454] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c30f49ccb1940908d297bbca090b0d5 in term 0.
I20260621 02:47:29.584985 23971 leader_election.cc:302] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c30f49ccb1940908d297bbca090b0d5, 77a4a103a8d645a6a75308c18cbb6e94; no voters:
I20260621 02:47:29.585337 24239 raft_consensus.cc:2781] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:47:29.585475 24239 raft_consensus.cc:482] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:47:29.585600 24239 raft_consensus.cc:3037] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:47:29.587966 24239 raft_consensus.cc:504] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 } } peers { permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 } } peers { permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 } }
I20260621 02:47:29.588501 24239 leader_election.cc:288] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [CANDIDATE]: Term 1 election: Requested vote from peers 77a4a103a8d645a6a75308c18cbb6e94 (127.9.166.3:34067), a68b070367da4cbcb3865225c1c034be (127.9.166.1:43803)
I20260621 02:47:29.588966 24169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86140a1bc7f64812b75dd0a490750077" candidate_uuid: "3c30f49ccb1940908d297bbca090b0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77a4a103a8d645a6a75308c18cbb6e94"
I20260621 02:47:29.589079 23902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86140a1bc7f64812b75dd0a490750077" candidate_uuid: "3c30f49ccb1940908d297bbca090b0d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68b070367da4cbcb3865225c1c034be"
I20260621 02:47:29.589169 24169 raft_consensus.cc:3037] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:47:29.589304 23902 raft_consensus.cc:3037] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:47:29.591660 24169 raft_consensus.cc:2454] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c30f49ccb1940908d297bbca090b0d5 in term 1.
I20260621 02:47:29.592067 23971 leader_election.cc:302] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c30f49ccb1940908d297bbca090b0d5, 77a4a103a8d645a6a75308c18cbb6e94; no voters:
I20260621 02:47:29.592085 23902 raft_consensus.cc:2454] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c30f49ccb1940908d297bbca090b0d5 in term 1.
I20260621 02:47:29.592345 24239 raft_consensus.cc:2781] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:47:29.592784 24239 raft_consensus.cc:686] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [term 1 LEADER]: Becoming Leader. State: Replica: 3c30f49ccb1940908d297bbca090b0d5, State: Running, Role: LEADER
I20260621 02:47:29.593250 24239 consensus_queue.cc:237] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 } } peers { permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 } } peers { permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 } }
I20260621 02:47:29.598381 23758 catalog_manager.cc:5495] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c30f49ccb1940908d297bbca090b0d5 (127.9.166.2). New cstate: current_term: 1 leader_uuid: "3c30f49ccb1940908d297bbca090b0d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 } health_report { overall_health: UNKNOWN } } }
W20260621 02:47:29.603946 23948 tablet.cc:2369] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:47:29.607559 24082 tablet.cc:2369] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:47:29.614439 24215 tablet.cc:2369] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:47:29.639247 24229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:29.634417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52590 (local address 127.9.166.2:36195)
0621 02:47:29.634597 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:29.634601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:29.634619 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:47:29.634700 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:29.634704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:29.634781 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:29.634907 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:47:29.635229 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.636033 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.637305 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.637687 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.637777 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:29.638210 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:47:29.638689 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:47:29.638793 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:29.638976 (+ 183us) server_negotiation.cc:299] Negotiation successful
0621 02:47:29.639038 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:47:29.647048 23902 raft_consensus.cc:1261] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be [term 1 FOLLOWER]: Refusing update from remote peer 3c30f49ccb1940908d297bbca090b0d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:47:29.647073 24169 raft_consensus.cc:1261] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94 [term 1 FOLLOWER]: Refusing update from remote peer 3c30f49ccb1940908d297bbca090b0d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:47:29.648023 24243 consensus_queue.cc:1035] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:47:29.648417 24239 consensus_queue.cc:1035] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:47:29.658504 24246 mvcc.cc:205] Tried to move back new op lower bound from 7299113163344883712 to 7299113163138260992. Current Snapshot: MvccSnapshot[applied={T|T < 7299113163344883712}]
I20260621 02:47:29.672783 23758 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:47:29.673098 23758 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:47:29.677008 23758 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569: Sending DeleteTablet for 3 replicas of tablet 86140a1bc7f64812b75dd0a490750077
I20260621 02:47:29.678186 24016 tablet_service.cc:1514] Processing DeleteTablet for tablet 86140a1bc7f64812b75dd0a490750077 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:47:29 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:52586
I20260621 02:47:29.678295 24149 tablet_service.cc:1514] Processing DeleteTablet for tablet 86140a1bc7f64812b75dd0a490750077 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:47:29 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:51734
I20260621 02:47:29.678918 23882 tablet_service.cc:1514] Processing DeleteTablet for tablet 86140a1bc7f64812b75dd0a490750077 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:47:29 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:34058
I20260621 02:47:29.679086 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 23819
I20260621 02:47:29.679222 24255 tablet_replica.cc:330] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5: stopping tablet replica
I20260621 02:47:29.679723 24255 raft_consensus.cc:2229] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:47:29.680440 24255 raft_consensus.cc:2258] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:47:29.682301 24255 ts_tablet_manager.cc:1905] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:47:29.689795 24255 ts_tablet_manager.cc:1918] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:47:29.690023 24255 log.cc:1196] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/86140a1bc7f64812b75dd0a490750077
I20260621 02:47:29.690702 24255 ts_tablet_manager.cc:1939] T 86140a1bc7f64812b75dd0a490750077 P 3c30f49ccb1940908d297bbca090b0d5: Deleting consensus metadata
I20260621 02:47:29.692731 23742 catalog_manager.cc:4841] TS 3c30f49ccb1940908d297bbca090b0d5 (127.9.166.2:36195): tablet 86140a1bc7f64812b75dd0a490750077 (table test-table [id=a1bc5d98180d47be90ef7c905bf81969]) successfully deleted
I20260621 02:47:29.693729 24257 tablet_replica.cc:330] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be: stopping tablet replica
I20260621 02:47:29.694303 24257 raft_consensus.cc:2229] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:47:29.694911 24257 raft_consensus.cc:2258] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:47:29.695936 24256 tablet_replica.cc:330] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94: stopping tablet replica
I20260621 02:47:29.696345 24256 raft_consensus.cc:2229] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:47:29.696827 24256 raft_consensus.cc:2258] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:47:29.699126 24256 ts_tablet_manager.cc:1905] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:47:29.700753 24257 ts_tablet_manager.cc:1905] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 21 02:47:29 dist-test-slave-hvgn krb5kdc[23713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:47:29.706682 24256 ts_tablet_manager.cc:1918] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:47:29.706832 24256 log.cc:1196] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/86140a1bc7f64812b75dd0a490750077
I20260621 02:47:29.707345 24256 ts_tablet_manager.cc:1939] T 86140a1bc7f64812b75dd0a490750077 P 77a4a103a8d645a6a75308c18cbb6e94: Deleting consensus metadata
I20260621 02:47:29.707693 24257 ts_tablet_manager.cc:1918] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:47:29.707870 24257 log.cc:1196] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/86140a1bc7f64812b75dd0a490750077
I20260621 02:47:29.708351 24257 ts_tablet_manager.cc:1939] T 86140a1bc7f64812b75dd0a490750077 P a68b070367da4cbcb3865225c1c034be: Deleting consensus metadata
I20260621 02:47:29.709432 23744 catalog_manager.cc:4841] TS 77a4a103a8d645a6a75308c18cbb6e94 (127.9.166.3:34067): tablet 86140a1bc7f64812b75dd0a490750077 (table test-table [id=a1bc5d98180d47be90ef7c905bf81969]) successfully deleted
I20260621 02:47:29.710085 23744 catalog_manager.cc:4841] TS a68b070367da4cbcb3865225c1c034be (127.9.166.1:43803): tablet 86140a1bc7f64812b75dd0a490750077 (table test-table [id=a1bc5d98180d47be90ef7c905bf81969]) successfully deleted
I20260621 02:47:29.714071 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:29.680392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34070 (local address 127.9.166.1:43803)
0621 02:47:29.680571 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:29.680590 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:29.689729 (+ 9139us) server_negotiation.cc:407] Connection header received
0621 02:47:29.690829 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:29.690838 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:29.690970 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:29.691073 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:29.696927 (+ 5854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.698047 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.700176 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.700872 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.701022 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:29.706791 (+ 5769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:29.706807 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:29.706825 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:29.706869 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:29.711108 (+ 4239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:29.712086 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:29.712094 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:29.712096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:29.712254 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:29.712671 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:29.712674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:29.712676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:29.713014 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:29.713243 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:29.713609 (+ 366us) server_negotiation.cc:299] Negotiation successful
0621 02:47:29.713772 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260621 02:47:29.714728 23942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:29.820497 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 23819
I20260621 02:47:29.839293 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 23953
Jun 21 02:47:29 dist-test-slave-hvgn krb5kdc[23713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.2@KRBTEST.COM
I20260621 02:47:29.857123 24229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:29.840123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.9.166.2:36195)
0621 02:47:29.840368 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:29.840374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:29.840396 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:47:29.840473 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:29.840483 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:29.840563 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:29.840672 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:29.841907 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.843109 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.844570 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.845244 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.845384 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:29.849828 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:29.849844 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:29.849864 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:29.849910 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:29.853885 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:29.854723 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:29.854731 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:29.854735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:29.854905 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:29.855312 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:29.855317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:29.855321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:29.855795 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:29.856131 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:29.856386 (+ 255us) server_negotiation.cc:299] Negotiation successful
0621 02:47:29.856592 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260621 02:47:29.857901 24076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:29.972995 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 23953
I20260621 02:47:29.991369 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 24086
Jun 21 02:47:29 dist-test-slave-hvgn krb5kdc[23713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.3@KRBTEST.COM
I20260621 02:47:30.006743 24228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:29.992191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51740 (local address 127.9.166.3:34067)
0621 02:47:29.992371 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:29.992378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:29.992475 (+ 97us) server_negotiation.cc:407] Connection header received
0621 02:47:29.992685 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:29.992689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:29.992776 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:29.992883 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:29.994160 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.994994 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.996526 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:29.996928 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:29.997059 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:30.001403 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:30.001417 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:30.001434 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:30.001465 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:30.004003 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:30.004679 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:30.004686 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:30.004689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:30.004847 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:30.005228 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:30.005236 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:30.005241 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:30.005572 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:30.005888 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:30.006185 (+ 297us) server_negotiation.cc:299] Negotiation successful
0621 02:47:30.006403 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 02:47:30.007368 24209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:30.094514 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 24086
I20260621 02:47:30.111306 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 23726
I20260621 02:47:30.112540 23788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:30.222254 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 23726
I20260621 02:47:30.243467 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:38157
--webserver_interface=127.9.166.62
--webserver_port=35895
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:38157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:47:30.564836 24263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:30.565160 24263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:30.565320 24263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:30.572685 24263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:47:30.572783 24263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:30.572916 24263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:30.573030 24263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:47:30.573117 24263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:47:30.583140 24263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:38157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:38157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=35895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:30.584455 24263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:30.585793 24263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:30.608258 24270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:30.608295 24269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:30.608335 24272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:30.609575 24263 server_base.cc:1034] running on GCE node
I20260621 02:47:30.610448 24263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:30.612121 24263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:30.613361 24263 hybrid_clock.cc:648] HybridClock initialized: now 1782010050613303 us; error 49 us; skew 500 ppm
I20260621 02:47:30.613756 24263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:30 dist-test-slave-hvgn krb5kdc[23713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010050, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:30.620134 24263 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:47:30.621893 24263 webserver.cc:460] Webserver started at http://127.9.166.62:35895/ using document root <none> and password file <none>
I20260621 02:47:30.622642 24263 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:30.622810 24263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:30.628643 24263 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 02:47:30.631577 24279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:30.632954 24263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:47:30.633177 24263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "7a60191689bc4a61ac3a97d4d5091569"
format_stamp: "Formatted at 2026-06-21 02:47:27 on dist-test-slave-hvgn"
server_key: "4b364f48819db996df814204054ff284"
server_key_iv: "47e72cd5cabbd3c02dbc3fcfe9b01f82"
server_key_version: "encryptionkey@0"
I20260621 02:47:30.634258 24263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:30.647151 24263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:30.651207 24263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:30.651499 24263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:30.670356 24263 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:38157
I20260621 02:47:30.670377 24330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:38157 every 8 connection(s)
I20260621 02:47:30.671914 24263 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:47:30.674569 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 24263
I20260621 02:47:30.675365 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:43803
--local_ip_for_outbound_sockets=127.9.166.1
--tserver_master_addrs=127.9.166.62:38157
--webserver_port=33165
--webserver_interface=127.9.166.1
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:47:30.679020 24331 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:47:30.682451 24331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569: Bootstrap starting.
I20260621 02:47:30.701959 24331 log.cc:824] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:30.713639 24331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:47:30.714187 24331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569: Bootstrap complete.
I20260621 02:47:30.723349 24331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60191689bc4a61ac3a97d4d5091569" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38157 } }
I20260621 02:47:30.724273 24331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a60191689bc4a61ac3a97d4d5091569, State: Initialized, Role: FOLLOWER
I20260621 02:47:30.724927 24331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60191689bc4a61ac3a97d4d5091569" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38157 } }
I20260621 02:47:30.725186 24331 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:47:30.725322 24331 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:47:30.725500 24331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 1 FOLLOWER]: Advancing to term 2
I20260621 02:47:30.729243 24331 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60191689bc4a61ac3a97d4d5091569" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38157 } }
I20260621 02:47:30.729745 24331 leader_election.cc:302] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a60191689bc4a61ac3a97d4d5091569; no voters:
I20260621 02:47:30.730381 24331 leader_election.cc:288] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 02:47:30.730512 24335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 2 FOLLOWER]: Leader election won for term 2
I20260621 02:47:30.731710 24335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 2 LEADER]: Becoming Leader. State: Replica: 7a60191689bc4a61ac3a97d4d5091569, State: Running, Role: LEADER
I20260621 02:47:30.732282 24335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60191689bc4a61ac3a97d4d5091569" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38157 } }
I20260621 02:47:30.732829 24331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:47:30.734889 24336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7a60191689bc4a61ac3a97d4d5091569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60191689bc4a61ac3a97d4d5091569" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38157 } } }
I20260621 02:47:30.735265 24336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [sys.catalog]: This master's current role is: LEADER
I20260621 02:47:30.735257 24337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a60191689bc4a61ac3a97d4d5091569. Latest consensus state: current_term: 2 leader_uuid: "7a60191689bc4a61ac3a97d4d5091569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60191689bc4a61ac3a97d4d5091569" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38157 } } }
I20260621 02:47:30.735967 24337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [sys.catalog]: This master's current role is: LEADER
I20260621 02:47:30.744722 24342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:47:30.752185 24342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:47:30.753549 24342 catalog_manager.cc:1259] Loaded cluster ID: 2e6276e78ee74f188bdcc84b93d976ed
I20260621 02:47:30.753661 24342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:47:30.756770 24342 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260621 02:47:30.774669 24342 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260621 02:47:30.789029 24342 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:47:30.790704 24342 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569: Loaded TSK: 0
I20260621 02:47:30.792266 24342 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 02:47:31.026683 24333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:31.026952 24333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:31.027097 24333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:31.034509 24333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:31.034611 24333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:31.034821 24333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:47:31.044792 24333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:43803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=33165
--tserver_master_addrs=127.9.166.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:31.046142 24333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:31.047513 24333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:31.062124 24358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:31.062144 24359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:31.063172 24333 server_base.cc:1034] running on GCE node
W20260621 02:47:31.063452 24361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:31.064234 24333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:31.065419 24333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:31.066679 24333 hybrid_clock.cc:648] HybridClock initialized: now 1782010051066627 us; error 59 us; skew 500 ppm
I20260621 02:47:31.067080 24333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:31 dist-test-slave-hvgn krb5kdc[23713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010051, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:31.071079 24333 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:47:31.072850 24333 webserver.cc:460] Webserver started at http://127.9.166.1:33165/ using document root <none> and password file <none>
I20260621 02:47:31.073779 24333 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:31.073931 24333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:31.080924 24333 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260621 02:47:31.083978 24368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:31.085486 24333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:47:31.085633 24333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "a68b070367da4cbcb3865225c1c034be"
format_stamp: "Formatted at 2026-06-21 02:47:28 on dist-test-slave-hvgn"
server_key: "3eec10c6e6d3f9330ab479dbb51e277e"
server_key_iv: "4289d9c704cfb78f61182b693e23a485"
server_key_version: "encryptionkey@0"
I20260621 02:47:31.086907 24333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:31.104803 24333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:31.110131 24333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:31.110404 24333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:31.111505 24333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:47:31.113165 24333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:47:31.113277 24333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:31.113386 24333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:47:31.113464 24333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:31.155543 24333 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:43803
I20260621 02:47:31.155629 24480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:43803 every 8 connection(s)
I20260621 02:47:31.156973 24333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:47:31.158237 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 24333
I20260621 02:47:31.158843 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:36195
--local_ip_for_outbound_sockets=127.9.166.2
--tserver_master_addrs=127.9.166.62:38157
--webserver_port=44679
--webserver_interface=127.9.166.2
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:47:31 dist-test-slave-hvgn krb5kdc[23713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010051, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:31.182273 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:31.160357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.1:41611)
0621 02:47:31.161187 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:31.161232 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:31.162919 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:31.164465 (+ 1546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:31.164473 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:31.164975 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:31.165495 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:31.165512 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:31.167728 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:31.167736 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:31.169918 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:31.169936 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:31.170933 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:31.170942 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:31.171266 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:31.172036 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:31.172071 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:31.174731 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:31.178273 (+ 3542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:31.178281 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:31.178296 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:31.178714 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:31.179169 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:31.179174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:31.179178 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:31.179357 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:31.180423 (+ 1066us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:31.180430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:31.180724 (+ 294us) client_negotiation.cc:769] Sending connection context
0621 02:47:31.180941 (+ 217us) client_negotiation.cc:240] Negotiation successful
0621 02:47:31.181305 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":346,"threads_started":1}
I20260621 02:47:31.183205 24483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:31.160022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:41611 (local address 127.9.166.62:38157)
0621 02:47:31.160691 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:31.160709 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:31.161378 (+ 669us) server_negotiation.cc:407] Connection header received
0621 02:47:31.163494 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:31.163514 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:31.163956 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:31.164368 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:31.165734 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:31.167564 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:31.170062 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:31.170699 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:31.170827 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:31.175010 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:31.175018 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:31.175033 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:31.175078 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:31.178128 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:31.178856 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:31.178870 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:31.178875 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:31.179038 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:31.179532 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:31.179541 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:31.179546 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:31.180281 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:31.180487 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:31.181242 (+ 755us) server_negotiation.cc:299] Negotiation successful
0621 02:47:31.181737 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":291,"threads_started":1}
I20260621 02:47:31.185976 24481 heartbeater.cc:347] Connected to a master server at 127.9.166.62:38157
I20260621 02:47:31.186283 24481 heartbeater.cc:464] Registering TS with master...
I20260621 02:47:31.187036 24481 heartbeater.cc:511] Master 127.9.166.62:38157 requested a full tablet report, sending...
I20260621 02:47:31.189126 24296 ts_manager.cc:194] Registered new tserver with Master: a68b070367da4cbcb3865225c1c034be (127.9.166.1:43803)
I20260621 02:47:31.190595 24296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:41611
W20260621 02:47:31.489559 24485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:31.489821 24485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:31.489966 24485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:31.497807 24485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:31.497916 24485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:31.498121 24485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:47:31.508256 24485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:36195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=44679
--tserver_master_addrs=127.9.166.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:31.509500 24485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:31.510792 24485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:31.523650 24490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:31.523684 24493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:31.523890 24491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:31.524523 24485 server_base.cc:1034] running on GCE node
I20260621 02:47:31.525108 24485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:31.526080 24485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:31.527308 24485 hybrid_clock.cc:648] HybridClock initialized: now 1782010051527277 us; error 49 us; skew 500 ppm
I20260621 02:47:31.527638 24485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:31 dist-test-slave-hvgn krb5kdc[23713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010051, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:31.530571 24485 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:47:31.531953 24485 webserver.cc:460] Webserver started at http://127.9.166.2:44679/ using document root <none> and password file <none>
I20260621 02:47:31.532712 24485 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:31.532864 24485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:31.538093 24485 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260621 02:47:31.540939 24500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:31.542269 24485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:47:31.542445 24485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "3c30f49ccb1940908d297bbca090b0d5"
format_stamp: "Formatted at 2026-06-21 02:47:28 on dist-test-slave-hvgn"
server_key: "aaebd01a5b7a68ef504e264e2b8aaaef"
server_key_iv: "893956cb596c04d73e60a6d2664b2d7c"
server_key_version: "encryptionkey@0"
I20260621 02:47:31.543480 24485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:31.559338 24485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:31.563707 24485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:31.564000 24485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:31.564996 24485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:47:31.566418 24485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:47:31.566550 24485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:31.566695 24485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:47:31.566790 24485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:31.605561 24485 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:36195
I20260621 02:47:31.605649 24612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:36195 every 8 connection(s)
I20260621 02:47:31.607023 24485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:47:31.614066 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 24485
I20260621 02:47:31.614735 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:34067
--local_ip_for_outbound_sockets=127.9.166.3
--tserver_master_addrs=127.9.166.62:38157
--webserver_port=37147
--webserver_interface=127.9.166.3
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:47:31 dist-test-slave-hvgn krb5kdc[23713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010051, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:31.628206 24483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:31.609724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:45609 (local address 127.9.166.62:38157)
0621 02:47:31.609911 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:31.609916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:31.611022 (+ 1106us) server_negotiation.cc:407] Connection header received
0621 02:47:31.612868 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:31.612872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:31.612966 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:31.613058 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:31.614681 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:31.615896 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:31.617421 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:31.618180 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:31.618330 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:31.622242 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:31.622246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:31.622250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:31.622285 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:31.625002 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:31.625903 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:31.625907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:31.625909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:31.626039 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:31.626518 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:31.626521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:31.626523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:31.626796 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:31.626906 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:31.627787 (+ 881us) server_negotiation.cc:299] Negotiation successful
0621 02:47:31.627962 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:47:31.629832 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:31.610166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.2:45609)
0621 02:47:31.610850 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:31.610886 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:31.612607 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:31.613308 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:31.613316 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:31.613865 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:31.614505 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:31.614525 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:31.616065 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:31.616069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:31.617266 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:31.617277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:31.618446 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:31.618454 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:31.618670 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:31.619372 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:31.619453 (+ 81us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:31.622015 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:31.625141 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:31.625152 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:31.625171 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:31.625786 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:31.626146 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:31.626153 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:31.626160 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:31.626408 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:31.626921 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:31.626932 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:31.627372 (+ 440us) client_negotiation.cc:769] Sending connection context
0621 02:47:31.627721 (+ 349us) client_negotiation.cc:240] Negotiation successful
0621 02:47:31.628354 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":267,"threads_started":1}
I20260621 02:47:31.631381 24613 heartbeater.cc:347] Connected to a master server at 127.9.166.62:38157
I20260621 02:47:31.631745 24613 heartbeater.cc:464] Registering TS with master...
I20260621 02:47:31.632472 24613 heartbeater.cc:511] Master 127.9.166.62:38157 requested a full tablet report, sending...
I20260621 02:47:31.634090 24296 ts_manager.cc:194] Registered new tserver with Master: 3c30f49ccb1940908d297bbca090b0d5 (127.9.166.2:36195)
I20260621 02:47:31.634902 24296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:45609
W20260621 02:47:31.945848 24616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:31.946127 24616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:31.946285 24616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:31.953626 24616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:31.953732 24616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:31.953922 24616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:47:31.963577 24616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:34067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=37147
--tserver_master_addrs=127.9.166.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:31.964812 24616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:31.966153 24616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:31.978838 24621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:31.978907 24624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:31.978814 24622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:31.980331 24616 server_base.cc:1034] running on GCE node
I20260621 02:47:31.980942 24616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:31.981880 24616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:31.983119 24616 hybrid_clock.cc:648] HybridClock initialized: now 1782010051983060 us; error 49 us; skew 500 ppm
I20260621 02:47:31.983484 24616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:31 dist-test-slave-hvgn krb5kdc[23713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010051, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:31.986490 24616 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:47:31.987993 24616 webserver.cc:460] Webserver started at http://127.9.166.3:37147/ using document root <none> and password file <none>
I20260621 02:47:31.988672 24616 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:31.988823 24616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:31.994179 24616 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:47:31.996871 24631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:31.998250 24616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260621 02:47:31.998432 24616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "77a4a103a8d645a6a75308c18cbb6e94"
format_stamp: "Formatted at 2026-06-21 02:47:29 on dist-test-slave-hvgn"
server_key: "aa4f49132c6e75f7c89f6753614eeb77"
server_key_iv: "105b0a06cb7cabca667668775b5227e7"
server_key_version: "encryptionkey@0"
I20260621 02:47:31.999424 24616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:32.023725 24616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:32.027997 24616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:32.028265 24616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:32.029266 24616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:47:32.030586 24616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:47:32.030715 24616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:32.030848 24616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:47:32.030946 24616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:32.064435 24616 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:34067
I20260621 02:47:32.064496 24743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:34067 every 8 connection(s)
I20260621 02:47:32.065868 24616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:47:32.070433 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 24616
Jun 21 02:47:32 dist-test-slave-hvgn krb5kdc[23713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010051, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:32.086757 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.071406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50424 (local address 127.9.166.62:38157)
0621 02:47:32.071956 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:32.071965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:32.072733 (+ 768us) server_negotiation.cc:407] Connection header received
0621 02:47:32.072829 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:32.072834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:32.072901 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:32.073020 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:32.074532 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.075466 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.076917 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.077514 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.077642 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:32.080407 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:32.080412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:32.080416 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:32.080445 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:32.083870 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:32.084711 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:32.084717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:32.084722 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:32.084892 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:32.085288 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:32.085294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:32.085297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:32.085560 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:32.085836 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:32.086134 (+ 298us) server_negotiation.cc:299] Negotiation successful
0621 02:47:32.086346 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":401,"spinlock_wait_cycles":51456,"thread_start_us":244,"threads_started":1}
I20260621 02:47:32.087081 24483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.068436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:55581 (local address 127.9.166.62:38157)
0621 02:47:32.068634 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:32.068638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:32.069678 (+ 1040us) server_negotiation.cc:407] Connection header received
0621 02:47:32.072181 (+ 2503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:32.072187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:32.072286 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:32.072430 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:32.074160 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.075040 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.076732 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.077357 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.077448 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:32.081142 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:32.081146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:32.081149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:32.081187 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:32.084007 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:32.084808 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:32.084817 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:32.084821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:32.084979 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:32.085422 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:32.085427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:32.085430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:32.085678 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:32.085829 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:32.086479 (+ 650us) server_negotiation.cc:299] Negotiation successful
0621 02:47:32.086677 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":82}
I20260621 02:47:32.088111 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.068823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.3:55581)
0621 02:47:32.069517 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:32.069561 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:32.071909 (+ 2348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:32.072787 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:32.072799 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:32.073436 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:32.073991 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:32.074009 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.075190 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.075196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:32.076592 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:32.076605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.077578 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.077587 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:32.077864 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:32.078656 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:32.078679 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:32.080906 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:32.084137 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:32.084143 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:32.084155 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:32.084680 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:32.085105 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:32.085110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:32.085112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:32.085289 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:32.085807 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:32.085815 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:32.086122 (+ 307us) client_negotiation.cc:769] Sending connection context
0621 02:47:32.086389 (+ 267us) client_negotiation.cc:240] Negotiation successful
0621 02:47:32.086880 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":286,"threads_started":1}
I20260621 02:47:32.089517 24744 heartbeater.cc:347] Connected to a master server at 127.9.166.62:38157
I20260621 02:47:32.089814 24744 heartbeater.cc:464] Registering TS with master...
I20260621 02:47:32.090466 24744 heartbeater.cc:511] Master 127.9.166.62:38157 requested a full tablet report, sending...
I20260621 02:47:32.091944 24296 ts_manager.cc:194] Registered new tserver with Master: 77a4a103a8d645a6a75308c18cbb6e94 (127.9.166.3:34067)
I20260621 02:47:32.092743 24296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:55581
I20260621 02:47:32.101483 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:47:32.108551 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.102495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50432 (local address 127.9.166.62:38157)
0621 02:47:32.102716 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:32.102721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:32.103229 (+ 508us) server_negotiation.cc:407] Connection header received
0621 02:47:32.103516 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:32.103520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:32.103721 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:32.103877 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:47:32.104349 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.105015 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.106458 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.106879 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.107017 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:32.107523 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:47:32.107994 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:47:32.108164 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:32.108213 (+ 49us) server_negotiation.cc:299] Negotiation successful
0621 02:47:32.108289 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260621 02:47:32.109436 24296 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:50432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:47:32.116532 24296 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:47:32.147444 24754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.127663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:43803 (local address 127.0.0.1:34084)
0621 02:47:32.135260 (+ 7597us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:32.135295 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:32.135534 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:32.136701 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:32.136712 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:32.136754 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:32.137304 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:32.137320 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.139872 (+ 2552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.139881 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:32.143149 (+ 3268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:32.143170 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.146069 (+ 2899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.146078 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:32.146353 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:32.146426 (+ 73us) client_negotiation.cc:769] Sending connection context
0621 02:47:32.146591 (+ 165us) client_negotiation.cc:240] Negotiation successful
0621 02:47:32.146871 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7327,"thread_start_us":7177,"threads_started":1}
I20260621 02:47:32.149152 24755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.127771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34084 (local address 127.9.166.1:43803)
0621 02:47:32.134741 (+ 6970us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:32.134756 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:32.135733 (+ 977us) server_negotiation.cc:407] Connection header received
0621 02:47:32.135846 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:32.135856 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:32.136155 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:32.136386 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:32.137556 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.139672 (+ 2116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.143396 (+ 3724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.145422 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.145583 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:32.145865 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:32.148465 (+ 2600us) server_negotiation.cc:299] Negotiation successful
0621 02:47:32.148717 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6742,"thread_start_us":6607,"threads_started":1}
I20260621 02:47:32.156237 24416 tablet_service.cc:1467] Processing CreateTablet for tablet f1071295d7b749b29c090d8d66c42d64 (DEFAULT_TABLE table=test-table [id=a5c0abd4801841b5b0b5455b395f58a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:47:32.157877 24754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.128090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:36195 (local address 127.0.0.1:52612)
0621 02:47:32.148107 (+ 20017us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:32.148133 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:32.148270 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:32.151588 (+ 3318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:32.151593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:32.151614 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:32.151792 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:32.151805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.153671 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.153677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:32.155263 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:32.155277 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.157096 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.157101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:32.157291 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:32.157336 (+ 45us) client_negotiation.cc:769] Sending connection context
0621 02:47:32.157405 (+ 69us) client_negotiation.cc:240] Negotiation successful
0621 02:47:32.157596 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19867,"thread_start_us":22816,"threads_started":1}
I20260621 02:47:32.158059 24416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1071295d7b749b29c090d8d66c42d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:47:32.158969 24757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.128358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.9.166.2:36195)
0621 02:47:32.150822 (+ 22464us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:32.150833 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:32.150876 (+ 43us) server_negotiation.cc:407] Connection header received
0621 02:47:32.151011 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:32.151023 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:32.151346 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:32.151984 (+ 638us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:32.152212 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.153524 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.155491 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.156916 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.158234 (+ 1318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:32.158440 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:32.158511 (+ 71us) server_negotiation.cc:299] Negotiation successful
0621 02:47:32.158673 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22264,"thread_start_us":22141,"threads_started":1}
I20260621 02:47:32.161108 24756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.135956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:34067 (local address 127.0.0.1:51756)
0621 02:47:32.151181 (+ 15225us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:32.151203 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:32.151324 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:32.152066 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:32.152072 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:32.152098 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:32.152392 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:32.152401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.154393 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.154398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:32.157362 (+ 2964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:32.157376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.160321 (+ 2945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.160326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:32.160542 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:32.160579 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 02:47:32.160677 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 02:47:32.160809 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15095,"thread_start_us":15983,"threads_started":1}
I20260621 02:47:32.161985 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.136024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51756 (local address 127.9.166.3:34067)
0621 02:47:32.150324 (+ 14300us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:32.150337 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:32.151464 (+ 1127us) server_negotiation.cc:407] Connection header received
0621 02:47:32.151577 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:32.151587 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:32.151840 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:32.152555 (+ 715us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:32.152762 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.154234 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.157547 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.160101 (+ 2554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.160273 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:32.160492 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:32.161455 (+ 963us) server_negotiation.cc:299] Negotiation successful
0621 02:47:32.161643 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14056,"thread_start_us":13859,"threads_started":1}
I20260621 02:47:32.164258 24548 tablet_service.cc:1467] Processing CreateTablet for tablet f1071295d7b749b29c090d8d66c42d64 (DEFAULT_TABLE table=test-table [id=a5c0abd4801841b5b0b5455b395f58a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:47:32.165619 24548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1071295d7b749b29c090d8d66c42d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:47:32.165361 24679 tablet_service.cc:1467] Processing CreateTablet for tablet f1071295d7b749b29c090d8d66c42d64 (DEFAULT_TABLE table=test-table [id=a5c0abd4801841b5b0b5455b395f58a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:47:32.166718 24679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1071295d7b749b29c090d8d66c42d64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:47:32.174074 24760 tablet_bootstrap.cc:492] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be: Bootstrap starting.
I20260621 02:47:32.177651 24760 tablet_bootstrap.cc:654] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be: Neither blocks nor log segments found. Creating new log.
I20260621 02:47:32.178637 24760 log.cc:824] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:32.180956 24762 tablet_bootstrap.cc:492] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5: Bootstrap starting.
I20260621 02:47:32.181973 24763 tablet_bootstrap.cc:492] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94: Bootstrap starting.
I20260621 02:47:32.182096 24760 tablet_bootstrap.cc:492] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be: No bootstrap required, opened a new log
I20260621 02:47:32.182363 24760 ts_tablet_manager.cc:1397] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260621 02:47:32.184967 24762 tablet_bootstrap.cc:654] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5: Neither blocks nor log segments found. Creating new log.
I20260621 02:47:32.185029 24763 tablet_bootstrap.cc:654] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94: Neither blocks nor log segments found. Creating new log.
I20260621 02:47:32.185966 24763 log.cc:824] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:32.186157 24762 log.cc:824] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:32.188720 24762 tablet_bootstrap.cc:492] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5: No bootstrap required, opened a new log
I20260621 02:47:32.188941 24763 tablet_bootstrap.cc:492] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94: No bootstrap required, opened a new log
I20260621 02:47:32.188980 24762 ts_tablet_manager.cc:1397] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 02:47:32.189183 24763 ts_tablet_manager.cc:1397] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260621 02:47:32.192999 24760 raft_consensus.cc:348] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 } } peers { permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 } } peers { permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 } }
I20260621 02:47:32.193434 24760 raft_consensus.cc:374] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:47:32.193529 24760 raft_consensus.cc:729] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68b070367da4cbcb3865225c1c034be, State: Initialized, Role: FOLLOWER
I20260621 02:47:32.194072 24481 heartbeater.cc:503] Master 127.9.166.62:38157 was elected leader, sending a full tablet report...
I20260621 02:47:32.194154 24760 consensus_queue.cc:260] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 } } peers { permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 } } peers { permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 } }
I20260621 02:47:32.197525 24762 raft_consensus.cc:348] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 } } peers { permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 } } peers { permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 } }
I20260621 02:47:32.197939 24762 raft_consensus.cc:374] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:47:32.198046 24762 raft_consensus.cc:729] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c30f49ccb1940908d297bbca090b0d5, State: Initialized, Role: FOLLOWER
I20260621 02:47:32.198639 24762 consensus_queue.cc:260] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 } } peers { permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 } } peers { permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 } }
I20260621 02:47:32.200232 24613 heartbeater.cc:503] Master 127.9.166.62:38157 was elected leader, sending a full tablet report...
I20260621 02:47:32.200806 24762 ts_tablet_manager.cc:1428] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260621 02:47:32.201663 24763 raft_consensus.cc:348] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 } } peers { permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 } } peers { permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 } }
I20260621 02:47:32.202087 24763 raft_consensus.cc:374] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:47:32.202214 24763 raft_consensus.cc:729] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77a4a103a8d645a6a75308c18cbb6e94, State: Initialized, Role: FOLLOWER
I20260621 02:47:32.202781 24763 consensus_queue.cc:260] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 } } peers { permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 } } peers { permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 } }
I20260621 02:47:32.203047 24760 ts_tablet_manager.cc:1428] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be: Time spent starting tablet: real 0.021s user 0.020s sys 0.000s
I20260621 02:47:32.204572 24744 heartbeater.cc:503] Master 127.9.166.62:38157 was elected leader, sending a full tablet report...
I20260621 02:47:32.205250 24763 ts_tablet_manager.cc:1428] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
W20260621 02:47:32.317518 24745 tablet.cc:2369] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:47:32.359127 24614 tablet.cc:2369] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:47:32.409392 24482 tablet.cc:2369] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:47:32.441792 24768 raft_consensus.cc:482] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:47:32.442116 24768 raft_consensus.cc:504] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 } } peers { permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 } } peers { permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 } }
I20260621 02:47:32.443886 24768 leader_election.cc:288] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a68b070367da4cbcb3865225c1c034be (127.9.166.1:43803), 3c30f49ccb1940908d297bbca090b0d5 (127.9.166.2:36195)
I20260621 02:47:32.450675 24757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.444329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:40411 (local address 127.9.166.2:36195)
0621 02:47:32.444575 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:32.444581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:32.444735 (+ 154us) server_negotiation.cc:407] Connection header received
0621 02:47:32.445005 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:32.445011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:32.445139 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:32.445233 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:32.445726 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.446683 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.448946 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.449743 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.449896 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:32.449999 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:32.450300 (+ 301us) server_negotiation.cc:299] Negotiation successful
0621 02:47:32.450432 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260621 02:47:32.450695 24746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.444228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:36195 (local address 127.9.166.3:40411)
0621 02:47:32.444604 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:32.444629 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:32.444801 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:32.445300 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:32.445307 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:32.445335 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:32.445565 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:32.445577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.446803 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.446808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:32.448757 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:32.448770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.449913 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.449919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:32.450114 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:32.450161 (+ 47us) client_negotiation.cc:769] Sending connection context
0621 02:47:32.450274 (+ 113us) client_negotiation.cc:240] Negotiation successful
0621 02:47:32.450406 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":21}
I20260621 02:47:32.450851 24755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.444460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:41017 (local address 127.9.166.1:43803)
0621 02:47:32.444677 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:32.444685 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:32.445228 (+ 543us) server_negotiation.cc:407] Connection header received
0621 02:47:32.445346 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:32.445351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:32.445428 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:32.445529 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:32.446087 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.446911 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.449072 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.449884 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.450070 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:32.450167 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:32.450458 (+ 291us) server_negotiation.cc:299] Negotiation successful
0621 02:47:32.450588 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 02:47:32.451016 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.444285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:43803 (local address 127.9.166.3:41017)
0621 02:47:32.445075 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:32.445098 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:32.445233 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:32.445549 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:32.445552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:32.445570 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:32.445929 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:32.445940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.447084 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.447088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:32.448910 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:32.448927 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.450119 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.450125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:32.450304 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:32.450327 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 02:47:32.450433 (+ 106us) client_negotiation.cc:240] Negotiation successful
0621 02:47:32.450568 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":41,"thread_start_us":454,"threads_started":1}
I20260621 02:47:32.451812 24568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1071295d7b749b29c090d8d66c42d64" candidate_uuid: "77a4a103a8d645a6a75308c18cbb6e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c30f49ccb1940908d297bbca090b0d5" is_pre_election: true
I20260621 02:47:32.452006 24436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1071295d7b749b29c090d8d66c42d64" candidate_uuid: "77a4a103a8d645a6a75308c18cbb6e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68b070367da4cbcb3865225c1c034be" is_pre_election: true
I20260621 02:47:32.452306 24568 raft_consensus.cc:2454] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77a4a103a8d645a6a75308c18cbb6e94 in term 0.
I20260621 02:47:32.452595 24436 raft_consensus.cc:2454] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77a4a103a8d645a6a75308c18cbb6e94 in term 0.
I20260621 02:47:32.453060 24632 leader_election.cc:302] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c30f49ccb1940908d297bbca090b0d5, 77a4a103a8d645a6a75308c18cbb6e94; no voters:
I20260621 02:47:32.453491 24768 raft_consensus.cc:2781] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:47:32.453629 24768 raft_consensus.cc:482] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:47:32.453754 24768 raft_consensus.cc:3037] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:47:32.456038 24768 raft_consensus.cc:504] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 } } peers { permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 } } peers { permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 } }
I20260621 02:47:32.456691 24768 leader_election.cc:288] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [CANDIDATE]: Term 1 election: Requested vote from peers a68b070367da4cbcb3865225c1c034be (127.9.166.1:43803), 3c30f49ccb1940908d297bbca090b0d5 (127.9.166.2:36195)
I20260621 02:47:32.457073 24436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1071295d7b749b29c090d8d66c42d64" candidate_uuid: "77a4a103a8d645a6a75308c18cbb6e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a68b070367da4cbcb3865225c1c034be"
I20260621 02:47:32.457243 24568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1071295d7b749b29c090d8d66c42d64" candidate_uuid: "77a4a103a8d645a6a75308c18cbb6e94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c30f49ccb1940908d297bbca090b0d5"
I20260621 02:47:32.457314 24436 raft_consensus.cc:3037] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:47:32.457463 24568 raft_consensus.cc:3037] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:47:32.459630 24436 raft_consensus.cc:2454] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77a4a103a8d645a6a75308c18cbb6e94 in term 1.
I20260621 02:47:32.460016 24568 raft_consensus.cc:2454] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77a4a103a8d645a6a75308c18cbb6e94 in term 1.
I20260621 02:47:32.460083 24634 leader_election.cc:302] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77a4a103a8d645a6a75308c18cbb6e94, a68b070367da4cbcb3865225c1c034be; no voters:
I20260621 02:47:32.460431 24768 raft_consensus.cc:2781] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:47:32.460855 24768 raft_consensus.cc:686] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [term 1 LEADER]: Becoming Leader. State: Replica: 77a4a103a8d645a6a75308c18cbb6e94, State: Running, Role: LEADER
I20260621 02:47:32.461318 24768 consensus_queue.cc:237] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 } } peers { permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 } } peers { permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 } }
I20260621 02:47:32.465867 24295 catalog_manager.cc:5495] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77a4a103a8d645a6a75308c18cbb6e94 (127.9.166.3). New cstate: current_term: 1 leader_uuid: "77a4a103a8d645a6a75308c18cbb6e94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77a4a103a8d645a6a75308c18cbb6e94" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 34067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 } health_report { overall_health: UNKNOWN } } }
I20260621 02:47:32.532133 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.526900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34130 (local address 127.9.166.3:34067)
0621 02:47:32.527139 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:32.527144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:32.527161 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:47:32.527417 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:32.527425 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:32.527512 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:32.527633 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:47:32.527959 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.528671 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.530049 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.530448 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.530544 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:32.530974 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:47:32.531543 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:47:32.531652 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:32.531756 (+ 104us) server_negotiation.cc:299] Negotiation successful
0621 02:47:32.531823 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260621 02:47:32.540125 24568 raft_consensus.cc:1261] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5 [term 1 FOLLOWER]: Refusing update from remote peer 77a4a103a8d645a6a75308c18cbb6e94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:47:32.540126 24436 raft_consensus.cc:1261] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be [term 1 FOLLOWER]: Refusing update from remote peer 77a4a103a8d645a6a75308c18cbb6e94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:47:32.541075 24768 consensus_queue.cc:1035] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c30f49ccb1940908d297bbca090b0d5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:47:32.541354 24770 consensus_queue.cc:1035] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a68b070367da4cbcb3865225c1c034be" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 43803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:47:32.567095 24296 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:50432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:47:32.567361 24296 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:50432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:47:32.570748 24296 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569: Sending DeleteTablet for 3 replicas of tablet f1071295d7b749b29c090d8d66c42d64
I20260621 02:47:32.571952 24416 tablet_service.cc:1514] Processing DeleteTablet for tablet f1071295d7b749b29c090d8d66c42d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:47:32 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:34084
I20260621 02:47:32.571944 24548 tablet_service.cc:1514] Processing DeleteTablet for tablet f1071295d7b749b29c090d8d66c42d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:47:32 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:52612
I20260621 02:47:32.571970 24679 tablet_service.cc:1514] Processing DeleteTablet for tablet f1071295d7b749b29c090d8d66c42d64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:47:32 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:51756
I20260621 02:47:32.573383 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 24333
I20260621 02:47:32.586493 24784 tablet_replica.cc:330] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5: stopping tablet replica
I20260621 02:47:32.587064 24783 tablet_replica.cc:330] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be: stopping tablet replica
I20260621 02:47:32.587798 24783 raft_consensus.cc:2229] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:47:32.588367 24784 raft_consensus.cc:2229] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:47:32.589497 24784 raft_consensus.cc:2258] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:47:32.588965 24783 raft_consensus.cc:2258] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:47:32.590641 24785 tablet_replica.cc:330] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94: stopping tablet replica
I20260621 02:47:32.591444 24785 raft_consensus.cc:2229] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:47:32.592278 24785 raft_consensus.cc:2258] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:47:32.592355 24783 ts_tablet_manager.cc:1905] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:47:32.593001 24784 ts_tablet_manager.cc:1905] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:47:32.597900 24755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.574401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35262 (local address 127.9.166.1:43803)
0621 02:47:32.575112 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:32.575121 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:32.575143 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:47:32.575209 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:32.575217 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:32.575329 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:32.575517 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:32.578065 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.579615 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.582325 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.583208 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.583351 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:32.588825 (+ 5474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:32.588836 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:32.588864 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:32.588910 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:32.593592 (+ 4682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:32.594484 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:32.594494 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:32.594498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:32.594695 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:32.595188 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:32.595194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:32.595198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:32.595702 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:32.597055 (+ 1353us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:32.597226 (+ 171us) server_negotiation.cc:299] Negotiation successful
0621 02:47:32.597491 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551}
I20260621 02:47:32.599536 24785 ts_tablet_manager.cc:1905] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:47:32.600044 24784 ts_tablet_manager.cc:1918] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:47:32.600266 24784 log.cc:1196] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/f1071295d7b749b29c090d8d66c42d64
I20260621 02:47:32.600373 24476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:32.600924 24784 ts_tablet_manager.cc:1939] T f1071295d7b749b29c090d8d66c42d64 P 3c30f49ccb1940908d297bbca090b0d5: Deleting consensus metadata
I20260621 02:47:32.602886 24280 catalog_manager.cc:4841] TS 3c30f49ccb1940908d297bbca090b0d5 (127.9.166.2:36195): tablet f1071295d7b749b29c090d8d66c42d64 (table test-table [id=a5c0abd4801841b5b0b5455b395f58a9]) successfully deleted
I20260621 02:47:32.600044 24783 ts_tablet_manager.cc:1918] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:47:32.603559 24783 log.cc:1196] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/f1071295d7b749b29c090d8d66c42d64
I20260621 02:47:32.608125 24785 ts_tablet_manager.cc:1918] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:47:32.608337 24785 log.cc:1196] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/f1071295d7b749b29c090d8d66c42d64
I20260621 02:47:32.608937 24785 ts_tablet_manager.cc:1939] T f1071295d7b749b29c090d8d66c42d64 P 77a4a103a8d645a6a75308c18cbb6e94: Deleting consensus metadata
I20260621 02:47:32.610747 24282 catalog_manager.cc:4841] TS 77a4a103a8d645a6a75308c18cbb6e94 (127.9.166.3:34067): tablet f1071295d7b749b29c090d8d66c42d64 (table test-table [id=a5c0abd4801841b5b0b5455b395f58a9]) successfully deleted
I20260621 02:47:32.707626 24783 ts_tablet_manager.cc:1939] T f1071295d7b749b29c090d8d66c42d64 P a68b070367da4cbcb3865225c1c034be: Deleting consensus metadata
I20260621 02:47:32.707829 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 24333
W20260621 02:47:32.725536 24282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.166.1:43803 (error 108)
I20260621 02:47:32.725911 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 24485
I20260621 02:47:32.728088 24788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.727190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:43803 (local address 127.0.0.1:35274)
0621 02:47:32.727749 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:32.727883 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":346,"threads_started":1}
W20260621 02:47:32.728426 24282 catalog_manager.cc:4584] TS a68b070367da4cbcb3865225c1c034be (127.9.166.1:43803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1071295d7b749b29c090d8d66c42d64: Network error: Client connection negotiation failed: client connection to 127.9.166.1:43803: connect: Connection refused (error 111)
I20260621 02:47:32.747608 24757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.726715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47366 (local address 127.9.166.2:36195)
0621 02:47:32.726943 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:32.726950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:32.727261 (+ 311us) server_negotiation.cc:407] Connection header received
0621 02:47:32.727658 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:32.727664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:32.727758 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:32.727882 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:32.729689 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.730404 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.732331 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.732817 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.732917 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:32.741402 (+ 8485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:32.741419 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:32.741434 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:32.741472 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:32.744483 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:32.745334 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:32.745341 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:32.745343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:32.745472 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:32.745873 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:32.745877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:32.745878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:32.746311 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:32.746550 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:32.746948 (+ 398us) server_negotiation.cc:299] Negotiation successful
0621 02:47:32.747187 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 02:47:32.748378 24608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:32.782356 24788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.781788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:43803 (local address 127.0.0.1:35280)
0621 02:47:32.782060 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:32.782178 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260621 02:47:32.783599 24788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.783129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:43803 (local address 127.0.0.1:35288)
0621 02:47:32.783319 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:32.783442 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260621 02:47:32.843290 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 24485
I20260621 02:47:32.844641 24788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.844070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:43803 (local address 127.0.0.1:35296)
0621 02:47:32.844323 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:32.844449 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260621 02:47:32.845795 24788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.845353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:43803 (local address 127.0.0.1:35300)
0621 02:47:32.845533 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:32.845626 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260621 02:47:32.862322 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 24616
I20260621 02:47:32.878782 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.863146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34146 (local address 127.9.166.3:34067)
0621 02:47:32.863339 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:32.863347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:32.863623 (+ 276us) server_negotiation.cc:407] Connection header received
0621 02:47:32.863809 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:32.863813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:32.863879 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:32.863977 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:32.865814 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.866647 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.868688 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:32.869130 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:32.869230 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:32.872603 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:32.872614 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:32.872629 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:32.872678 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:32.875783 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:32.876703 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:32.876710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:32.876713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:32.876872 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:32.877300 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:32.877304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:32.877306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:32.877662 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:32.877907 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:32.878289 (+ 382us) server_negotiation.cc:299] Negotiation successful
0621 02:47:32.878448 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260621 02:47:32.879590 24739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:32.926476 24788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.925984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:43803 (local address 127.0.0.1:35310)
0621 02:47:32.926229 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:32.926323 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260621 02:47:32.927740 24788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:32.927321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:43803 (local address 127.0.0.1:35320)
0621 02:47:32.927547 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:32.927616 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260621 02:47:33.002437 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 24616
I20260621 02:47:33.021960 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 24263
I20260621 02:47:33.023079 24326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:33.135725 24788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:33.135142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:43803 (local address 127.0.0.1:35322)
0621 02:47:33.135342 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:33.135493 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260621 02:47:33.135957 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 24263
I20260621 02:47:33.153751 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:38157
--webserver_interface=127.9.166.62
--webserver_port=35895
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:38157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:47:33.476752 24792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:33.477056 24792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:33.477146 24792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:33.484346 24792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:47:33.484459 24792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:33.484505 24792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:33.484562 24792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:47:33.484606 24792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:47:33.494546 24792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:38157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:38157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=35895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:33.496028 24792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:33.497327 24792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:33.511642 24800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:33.511695 24798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:33.511747 24797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:33.512639 24792 server_base.cc:1034] running on GCE node
I20260621 02:47:33.513481 24792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:33.514878 24792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:33.516091 24792 hybrid_clock.cc:648] HybridClock initialized: now 1782010053516033 us; error 41 us; skew 500 ppm
I20260621 02:47:33.516469 24792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:33 dist-test-slave-hvgn krb5kdc[23713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010053, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:33.519603 24792 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:47:33.521051 24792 webserver.cc:460] Webserver started at http://127.9.166.62:35895/ using document root <none> and password file <none>
I20260621 02:47:33.521679 24792 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:33.521786 24792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:33.526971 24792 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:47:33.529754 24807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:33.531031 24792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 02:47:33.531204 24792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "7a60191689bc4a61ac3a97d4d5091569"
format_stamp: "Formatted at 2026-06-21 02:47:27 on dist-test-slave-hvgn"
server_key: "4b364f48819db996df814204054ff284"
server_key_iv: "47e72cd5cabbd3c02dbc3fcfe9b01f82"
server_key_version: "encryptionkey@0"
I20260621 02:47:33.532224 24792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:33.562387 24792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:33.566792 24792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:33.567085 24792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:33.586611 24792 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:38157
I20260621 02:47:33.586622 24858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:38157 every 8 connection(s)
I20260621 02:47:33.588529 24792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:47:33.594874 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 24792
I20260621 02:47:33.595590 24859 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:47:33.595954 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:43803
--local_ip_for_outbound_sockets=127.9.166.1
--tserver_master_addrs=127.9.166.62:38157
--webserver_port=33165
--webserver_interface=127.9.166.1
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:47:33.600656 24859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569: Bootstrap starting.
I20260621 02:47:33.621691 24859 log.cc:824] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:33.638037 24859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:47:33.638553 24859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569: Bootstrap complete.
I20260621 02:47:33.647693 24859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60191689bc4a61ac3a97d4d5091569" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38157 } }
I20260621 02:47:33.648562 24859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a60191689bc4a61ac3a97d4d5091569, State: Initialized, Role: FOLLOWER
I20260621 02:47:33.649215 24859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60191689bc4a61ac3a97d4d5091569" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38157 } }
I20260621 02:47:33.649464 24859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:47:33.649598 24859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:47:33.649809 24859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 2 FOLLOWER]: Advancing to term 3
I20260621 02:47:33.653559 24859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60191689bc4a61ac3a97d4d5091569" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38157 } }
I20260621 02:47:33.654050 24859 leader_election.cc:302] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a60191689bc4a61ac3a97d4d5091569; no voters:
I20260621 02:47:33.654745 24859 leader_election.cc:288] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260621 02:47:33.654886 24863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 3 FOLLOWER]: Leader election won for term 3
I20260621 02:47:33.656003 24863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [term 3 LEADER]: Becoming Leader. State: Replica: 7a60191689bc4a61ac3a97d4d5091569, State: Running, Role: LEADER
I20260621 02:47:33.656620 24863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60191689bc4a61ac3a97d4d5091569" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38157 } }
I20260621 02:47:33.656960 24859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:47:33.659276 24865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a60191689bc4a61ac3a97d4d5091569. Latest consensus state: current_term: 3 leader_uuid: "7a60191689bc4a61ac3a97d4d5091569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60191689bc4a61ac3a97d4d5091569" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38157 } } }
I20260621 02:47:33.659365 24864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7a60191689bc4a61ac3a97d4d5091569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a60191689bc4a61ac3a97d4d5091569" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 38157 } } }
I20260621 02:47:33.659696 24865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [sys.catalog]: This master's current role is: LEADER
I20260621 02:47:33.659750 24864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a60191689bc4a61ac3a97d4d5091569 [sys.catalog]: This master's current role is: LEADER
I20260621 02:47:33.660576 24869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:47:33.683439 24869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:47:33.685468 24869 catalog_manager.cc:1259] Loaded cluster ID: 2e6276e78ee74f188bdcc84b93d976ed
I20260621 02:47:33.685688 24869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260621 02:47:33.688314 24869 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc12d6509ad google::LogMessage::Fail() at ??:0
@ 0x7fc12d65388a google::LogMessage::SendToLog() at ??:0
@ 0x7fc12d6504de google::LogMessage::Flush() at ??:0
@ 0x7fc12d651a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260621 02:47:33.709954 24881 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fc143083512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fc143080976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc143108610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fc143108439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc13029c345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc13029f91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fc13029f799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fc13027aeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc1335546db start_thread at ??:0
@ 0x7fc12c4ea61f clone at ??:0
W20260621 02:47:33.956511 24861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:33.956835 24861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:33.957002 24861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:33.964972 24861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:33.965140 24861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:33.965328 24861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:47:33.975497 24861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:43803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=33165
--tserver_master_addrs=127.9.166.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:33.976830 24861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:33.978127 24861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:33.991653 24903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:33.991719 24901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:33.991689 24900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:33.992424 24861 server_base.cc:1034] running on GCE node
I20260621 02:47:33.993037 24861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:33.993994 24861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:33.995328 24861 hybrid_clock.cc:648] HybridClock initialized: now 1782010053995233 us; error 66 us; skew 500 ppm
I20260621 02:47:33.995705 24861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:33 dist-test-slave-hvgn krb5kdc[23713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010053, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:33.998893 24861 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:47:34.000449 24861 webserver.cc:460] Webserver started at http://127.9.166.1:33165/ using document root <none> and password file <none>
I20260621 02:47:34.001183 24861 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:34.001331 24861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:34.006640 24861 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.003s
I20260621 02:47:34.009438 24910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:34.010886 24861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:47:34.011092 24861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "a68b070367da4cbcb3865225c1c034be"
format_stamp: "Formatted at 2026-06-21 02:47:28 on dist-test-slave-hvgn"
server_key: "3eec10c6e6d3f9330ab479dbb51e277e"
server_key_iv: "4289d9c704cfb78f61182b693e23a485"
server_key_version: "encryptionkey@0"
I20260621 02:47:34.012293 24861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:34.040011 24861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:34.044183 24861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:34.044409 24861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:34.045313 24861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:47:34.046540 24861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:47:34.046643 24861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:34.046728 24861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:47:34.046782 24861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:34.080636 24861 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:43803
I20260621 02:47:34.080823 25022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:43803 every 8 connection(s)
I20260621 02:47:34.082135 24861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:47:34.086931 25025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:34.085089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.1:40161)
0621 02:47:34.085840 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:34.086068 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":281,"threads_started":1}
W20260621 02:47:34.087699 25023 heartbeater.cc:644] Failed to heartbeat to 127.9.166.62:38157 (0 consecutive failures): Network error: Failed to ping master at 127.9.166.62:38157: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
I20260621 02:47:34.088488 25025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:34.088183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.1:47235)
0621 02:47:34.088320 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:34.088385 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260621 02:47:34.089365 25025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:34.089022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.1:55617)
0621 02:47:34.089172 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:34.089256 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260621 02:47:34.089665 25023 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 02:47:34.091466 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 24861
I20260621 02:47:34.092047 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:36195
--local_ip_for_outbound_sockets=127.9.166.2
--tserver_master_addrs=127.9.166.62:38157
--webserver_port=44679
--webserver_interface=127.9.166.2
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:47:34.420429 25026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:34.420706 25026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:34.420799 25026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:34.427894 25026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:34.427989 25026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:34.428097 25026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:47:34.437536 25026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:36195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=44679
--tserver_master_addrs=127.9.166.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:34.438675 25026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:34.439973 25026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:34.452879 25031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:34.452849 25032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:34.452940 25034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:34.453816 25026 server_base.cc:1034] running on GCE node
I20260621 02:47:34.454473 25026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:34.455433 25026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:34.456672 25026 hybrid_clock.cc:648] HybridClock initialized: now 1782010054456621 us; error 46 us; skew 500 ppm
I20260621 02:47:34.456984 25026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:34 dist-test-slave-hvgn krb5kdc[23713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010054, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:34.460199 25026 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:47:34.461704 25026 webserver.cc:460] Webserver started at http://127.9.166.2:44679/ using document root <none> and password file <none>
I20260621 02:47:34.462445 25026 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:34.462612 25026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:34.468477 25026 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 02:47:34.471349 25041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:34.472779 25026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:47:34.472975 25026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "3c30f49ccb1940908d297bbca090b0d5"
format_stamp: "Formatted at 2026-06-21 02:47:28 on dist-test-slave-hvgn"
server_key: "aaebd01a5b7a68ef504e264e2b8aaaef"
server_key_iv: "893956cb596c04d73e60a6d2664b2d7c"
server_key_version: "encryptionkey@0"
I20260621 02:47:34.473969 25026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:34.488209 25026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:34.492025 25026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:34.492298 25026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:34.493193 25026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:47:34.494455 25026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:47:34.494611 25026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:34.494736 25026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:47:34.494874 25026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260621 02:47:34.528702 25026 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:36195
I20260621 02:47:34.528789 25153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:36195 every 8 connection(s)
I20260621 02:47:34.530148 25026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:47:34.534823 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:34.533101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.2:50095)
0621 02:47:34.533792 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:34.533996 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":257,"threads_started":1}
W20260621 02:47:34.535701 25154 heartbeater.cc:644] Failed to heartbeat to 127.9.166.62:38157 (0 consecutive failures): Network error: Failed to ping master at 127.9.166.62:38157: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
I20260621 02:47:34.535903 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 25026
I20260621 02:47:34.536548 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:34.536249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.2:44249)
0621 02:47:34.536385 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:34.536450 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260621 02:47:34.536664 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:34067
--local_ip_for_outbound_sockets=127.9.166.3
--tserver_master_addrs=127.9.166.62:38157
--webserver_port=37147
--webserver_interface=127.9.166.3
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:47:34.537500 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:34.537193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.2:39617)
0621 02:47:34.537319 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:34.537390 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 02:47:34.537794 25154 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 02:47:34.856240 25157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:34.856513 25157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:34.856609 25157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:34.864367 25157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:34.864476 25157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:34.864590 25157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:47:34.874158 25157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:34067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=37147
--tserver_master_addrs=127.9.166.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:34.875367 25157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:34.876669 25157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:34.889588 25165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:34.889670 25162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:34.889657 25163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:34.890642 25157 server_base.cc:1034] running on GCE node
I20260621 02:47:34.891252 25157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:34.892212 25157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:34.893445 25157 hybrid_clock.cc:648] HybridClock initialized: now 1782010054893409 us; error 51 us; skew 500 ppm
I20260621 02:47:34.893772 25157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:34 dist-test-slave-hvgn krb5kdc[23713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010054, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:34.897015 25157 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:47:34.898422 25157 webserver.cc:460] Webserver started at http://127.9.166.3:37147/ using document root <none> and password file <none>
I20260621 02:47:34.899053 25157 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:34.899150 25157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:34.904300 25157 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:47:34.906925 25172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:34.908380 25157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:47:34.908524 25157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "77a4a103a8d645a6a75308c18cbb6e94"
format_stamp: "Formatted at 2026-06-21 02:47:29 on dist-test-slave-hvgn"
server_key: "aa4f49132c6e75f7c89f6753614eeb77"
server_key_iv: "105b0a06cb7cabca667668775b5227e7"
server_key_version: "encryptionkey@0"
I20260621 02:47:34.909444 25157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:34.931241 25157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:34.935567 25157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:34.935863 25157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:34.936820 25157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:47:34.938138 25157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:47:34.938261 25157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:34.938383 25157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:47:34.938485 25157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:34.971271 25157 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:34067
I20260621 02:47:34.971365 25284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:34067 every 8 connection(s)
I20260621 02:47:34.972767 25157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:47:34.977579 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:34.975805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.3:35599)
0621 02:47:34.976530 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:34.976757 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":266,"threads_started":1}
W20260621 02:47:34.978286 25285 heartbeater.cc:644] Failed to heartbeat to 127.9.166.62:38157 (0 consecutive failures): Network error: Failed to ping master at 127.9.166.62:38157: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
I20260621 02:47:34.979056 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:34.978749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.3:33205)
0621 02:47:34.978887 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:34.978963 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 02:47:34.979288 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 25157
I20260621 02:47:34.979890 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:34.979568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.3:55529)
0621 02:47:34.979711 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:34.979784 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260621 02:47:34.980191 25285 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 02:47:35.091137 25289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:35.090221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.1:35893)
0621 02:47:35.090778 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:35.090882 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":301,"threads_started":1}
I20260621 02:47:35.539319 25290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:35.538528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.2:55021)
0621 02:47:35.538965 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:35.539105 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":212,"threads_started":1}
I20260621 02:47:35.981444 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:35.980716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.3:46169)
0621 02:47:35.981143 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:35.981227 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":210,"threads_started":1}
I20260621 02:47:36.092732 25292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:36.092044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.1:60953)
0621 02:47:36.092488 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:36.092573 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":231,"threads_started":1}
I20260621 02:47:36.540934 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:36.540226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.2:54415)
0621 02:47:36.540693 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:36.540774 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":240,"threads_started":1}
I20260621 02:47:36.983044 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:36.982318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.3:33601)
0621 02:47:36.982781 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:36.982858 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":249,"threads_started":1}
I20260621 02:47:37.094308 25295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:37.093529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.1:60869)
0621 02:47:37.094030 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:37.094134 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":212,"threads_started":1}
I20260621 02:47:37.542518 25296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:37.541802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.2:47889)
0621 02:47:37.542266 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:37.542347 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":244,"threads_started":1}
I20260621 02:47:37.984612 25297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:37.983946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.3:54753)
0621 02:47:37.984360 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:37.984440 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":218,"threads_started":1}
I20260621 02:47:38.096120 25298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:38.095281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.1:59481)
0621 02:47:38.095826 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:38.095917 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":292,"threads_started":1}
I20260621 02:47:38.544435 25299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:38.543495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.2:40103)
0621 02:47:38.544013 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:38.544129 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":212,"threads_started":1}
I20260621 02:47:38.986194 25300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:38.985470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.3:37993)
0621 02:47:38.985898 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:38.985978 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":225,"threads_started":1}
I20260621 02:47:39.097813 25301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:39.097013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.1:48327)
0621 02:47:39.097563 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:39.097647 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":311,"threads_started":1}
I20260621 02:47:39.546245 25302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:39.545470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.2:53703)
0621 02:47:39.545956 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:39.546044 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":234,"threads_started":1}
I20260621 02:47:39.987774 25303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:39.987013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.3:52997)
0621 02:47:39.987509 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:39.987610 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":234,"threads_started":1}
I20260621 02:47:40.099362 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:40.098698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.1:33057)
0621 02:47:40.099117 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:40.099204 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":216,"threads_started":1}
I20260621 02:47:40.547894 25305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:40.547138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.2:45547)
0621 02:47:40.547640 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:40.547730 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":277,"threads_started":1}
I20260621 02:47:40.989305 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:40.988647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.3:35659)
0621 02:47:40.989067 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:40.989151 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":212,"threads_started":1}
I20260621 02:47:41.101186 25307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:41.100409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.1:41261)
0621 02:47:41.100898 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:41.100979 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":211,"threads_started":1}
I20260621 02:47:41.549654 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:41.548807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.2:57021)
0621 02:47:41.549348 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:41.549430 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":213,"threads_started":1}
I20260621 02:47:41.990958 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:41.990176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.3:59339)
0621 02:47:41.990662 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:41.990743 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":231,"threads_started":1}
I20260621 02:47:42.102846 25310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:42.102141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.1:45505)
0621 02:47:42.102580 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:42.102658 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":239,"threads_started":1}
I20260621 02:47:42.551347 25311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:42.550603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.2:36521)
0621 02:47:42.551046 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:42.551130 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":226,"threads_started":1}
I20260621 02:47:42.992507 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:42.991846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.3:39617)
0621 02:47:42.992249 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:42.992332 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":212,"threads_started":1}
I20260621 02:47:43.104386 25313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:43.103701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.1:45047)
0621 02:47:43.104135 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:43.104210 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":215,"threads_started":1}
I20260621 02:47:43.553040 25314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:43.552262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.2:32979)
0621 02:47:43.552719 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:43.552845 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":214,"threads_started":1}
I20260621 02:47:43.994268 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:43.993476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.3:54969)
0621 02:47:43.993965 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:43.994067 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":243,"threads_started":1}
I20260621 02:47:44.106030 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:44.105311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.1:42829)
0621 02:47:44.105777 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:44.105867 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":250,"threads_started":1}
I20260621 02:47:44.554889 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:44.554148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.2:53505)
0621 02:47:44.554629 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:44.554726 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":256,"threads_started":1}
I20260621 02:47:44.996057 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:44.995278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.3:36697)
0621 02:47:44.995817 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:44.995898 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":287,"threads_started":1}
I20260621 02:47:45.107661 25319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:45.106881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.1:51393)
0621 02:47:45.107328 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:45.107488 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":244,"threads_started":1}
I20260621 02:47:45.556545 25320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:45.555862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.2:59937)
0621 02:47:45.556288 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:45.556371 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":214,"threads_started":1}
I20260621 02:47:45.997762 25321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:45.997025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.3:53923)
0621 02:47:45.997491 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:45.997585 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":226,"threads_started":1}
I20260621 02:47:46.109189 25322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:46.108547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.1:60049)
0621 02:47:46.108961 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:46.109039 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":208,"threads_started":1}
I20260621 02:47:46.558143 25323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:46.557468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.2:50821)
0621 02:47:46.557902 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:46.557986 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":223,"threads_started":1}
I20260621 02:47:46.999563 25324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:46.998691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.3:50671)
0621 02:47:46.999243 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:46.999326 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":300,"threads_started":1}
I20260621 02:47:47.110845 25325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:47.110050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.1:35963)
0621 02:47:47.110524 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:47.110617 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":249,"threads_started":1}
I20260621 02:47:47.559804 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:47.559023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.2:52971)
0621 02:47:47.559521 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:47.559621 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":215,"threads_started":1}
I20260621 02:47:48.001331 25327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:48.000428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.3:49447)
0621 02:47:48.001028 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:48.001129 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":320,"threads_started":1}
I20260621 02:47:48.112607 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:48.111772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.1:60877)
0621 02:47:48.112329 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:48.112409 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":298,"threads_started":1}
I20260621 02:47:48.561971 25329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:48.560887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.2:49175)
0621 02:47:48.561602 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:48.561711 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":286,"threads_started":1}
I20260621 02:47:49.002828 25330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:49.002140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.3:55299)
0621 02:47:49.002558 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:49.002638 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":218,"threads_started":1}
I20260621 02:47:49.114594 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:49.113823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.1:48343)
0621 02:47:49.114300 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:49.114413 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":252,"threads_started":1}
I20260621 02:47:49.563908 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:49.563148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.2:46695)
0621 02:47:49.563632 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:49.563727 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":250,"threads_started":1}
I20260621 02:47:49.989197 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 24861
I20260621 02:47:50.004871 25334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:50.004057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:38157 (local address 127.9.166.3:52999)
0621 02:47:50.004572 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:50.004684 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":282,"threads_started":1}
I20260621 02:47:50.008184 25333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:49.990537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40336 (local address 127.9.166.1:43803)
0621 02:47:49.991338 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:49.991362 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:49.991433 (+ 71us) server_negotiation.cc:407] Connection header received
0621 02:47:49.993107 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:49.993116 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:49.993559 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:49.993921 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:49.995436 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:49.996650 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:49.997896 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:49.998541 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:49.998758 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:50.001586 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:50.001594 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:50.001608 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:50.001647 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:50.004857 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:50.005597 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:50.005604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:50.005610 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:50.005770 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:50.006162 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:50.006169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:50.006172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:50.006683 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:50.006989 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:50.007490 (+ 501us) server_negotiation.cc:299] Negotiation successful
0621 02:47:50.007905 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":443,"threads_started":1}
I20260621 02:47:50.009595 25018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:50.092927 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 24861
I20260621 02:47:50.109611 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 25026
I20260621 02:47:50.127452 25336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:50.110470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.9.166.2:36195)
0621 02:47:50.110992 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:50.111006 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:50.111062 (+ 56us) server_negotiation.cc:407] Connection header received
0621 02:47:50.112880 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:50.112893 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:50.113372 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:50.113731 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:50.115438 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:50.116680 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:50.117926 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:50.118571 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:50.118712 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:50.121434 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:50.121442 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:50.121458 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:50.121495 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:50.124384 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:50.125021 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:50.125027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:50.125034 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:50.125214 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:50.125536 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:50.125539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:50.125541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:50.126042 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:50.126287 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:50.126630 (+ 343us) server_negotiation.cc:299] Negotiation successful
0621 02:47:50.127048 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":239,"threads_started":1}
I20260621 02:47:50.128854 25149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:50.212271 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 25026
I20260621 02:47:50.229781 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 25157
I20260621 02:47:50.250187 25338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:50.230733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46444 (local address 127.9.166.3:34067)
0621 02:47:50.231430 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:50.231452 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:50.231493 (+ 41us) server_negotiation.cc:407] Connection header received
0621 02:47:50.233125 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:50.233134 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:50.233590 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:50.233883 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:50.235239 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:50.236778 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:50.238077 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:50.239122 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:50.239337 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:50.242623 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:50.242633 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:50.242653 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:50.242695 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:50.246680 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:50.247593 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:50.247602 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:50.247611 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:50.247869 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:50.248288 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:50.248291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:50.248293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:50.248798 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:50.249009 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:50.249457 (+ 448us) server_negotiation.cc:299] Negotiation successful
0621 02:47:50.249875 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":315,"threads_started":1}
I20260621 02:47:50.251638 25280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:50.333887 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 25157
2026-06-21T02:47:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24433 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:47:50 dist-test-slave-hvgn krb5kdc[25342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:47:50 dist-test-slave-hvgn krb5kdc[25342](info): set up 2 sockets
Jun 21 02:47:50 dist-test-slave-hvgn krb5kdc[25342](info): commencing operation
krb5kdc: starting...
W20260621 02:47:51.436318 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:47:51 dist-test-slave-hvgn krb5kdc[25342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:47:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:47:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:47:51.664669 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:39557
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:39557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:47:51.987018 25355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:51.987344 25355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:51.987524 25355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:51.995265 25355 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 02:47:51.995362 25355 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 02:47:51.995451 25355 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 02:47:51.995520 25355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:47:51.995568 25355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:51.995633 25355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:51.995676 25355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:47:51.995728 25355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:47:52.005596 25355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:39557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:39557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:52.006812 25355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:52.008174 25355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:52.020392 25361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:52.020445 25360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:52.021051 25355 server_base.cc:1034] running on GCE node
W20260621 02:47:52.021222 25363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:52.022042 25355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:52.023523 25355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:52.024796 25355 hybrid_clock.cc:648] HybridClock initialized: now 1782010072024719 us; error 47 us; skew 500 ppm
I20260621 02:47:52.025197 25355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:52 dist-test-slave-hvgn krb5kdc[25342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010072, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:52.028426 25355 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:47:52.029835 25355 webserver.cc:460] Webserver started at http://127.9.166.62:43155/ using document root <none> and password file <none>
I20260621 02:47:52.030563 25355 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:52.030695 25355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:52.031033 25355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:47:52.034123 25355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c"
format_stamp: "Formatted at 2026-06-21 02:47:52 on dist-test-slave-hvgn"
server_key: "07184a62abd3b3472b81ba7c8159d7da"
server_key_iv: "0a85f39fd3133200e950f79d31052683"
server_key_version: "encryptionkey@0"
I20260621 02:47:52.034893 25355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c"
format_stamp: "Formatted at 2026-06-21 02:47:52 on dist-test-slave-hvgn"
server_key: "07184a62abd3b3472b81ba7c8159d7da"
server_key_iv: "0a85f39fd3133200e950f79d31052683"
server_key_version: "encryptionkey@0"
I20260621 02:47:52.039994 25355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260621 02:47:52.043241 25370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:52.044533 25355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:47:52.044715 25355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c"
format_stamp: "Formatted at 2026-06-21 02:47:52 on dist-test-slave-hvgn"
server_key: "07184a62abd3b3472b81ba7c8159d7da"
server_key_iv: "0a85f39fd3133200e950f79d31052683"
server_key_version: "encryptionkey@0"
I20260621 02:47:52.044934 25355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:52.060381 25355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:52.064312 25355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:52.064591 25355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:52.082993 25355 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:39557
I20260621 02:47:52.083011 25421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:39557 every 8 connection(s)
I20260621 02:47:52.084362 25355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:47:52.085549 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 25355
I20260621 02:47:52.085773 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:47:52.086432 9880 external_mini_cluster.cc:1428] Setting key 2d32604881f9996d01ab9056ab73fdf0
I20260621 02:47:52.088666 25422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:47:52 dist-test-slave-hvgn krb5kdc[25342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:52.102043 25422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c: Bootstrap starting.
I20260621 02:47:52.106305 25422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c: Neither blocks nor log segments found. Creating new log.
I20260621 02:47:52.107607 25422 log.cc:824] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:52.110726 25425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:52.088377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51330 (local address 127.9.166.62:39557)
0621 02:47:52.089148 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:52.089160 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:52.089199 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:47:52.090767 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:52.090787 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:52.091156 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:52.091539 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:52.093376 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:52.094661 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:52.096558 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:52.097239 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:52.097381 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:52.102131 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:52.102142 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:52.102162 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:52.102214 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:52.106103 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:52.106868 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:52.106876 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:52.106883 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:52.107073 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:52.107456 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:52.107461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:52.107464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:52.108187 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:52.108474 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:52.108967 (+ 493us) server_negotiation.cc:299] Negotiation successful
0621 02:47:52.109419 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":360,"threads_started":1}
I20260621 02:47:52.111097 25422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c: No bootstrap required, opened a new log
I20260621 02:47:52.118980 25422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39557 } }
I20260621 02:47:52.119318 25422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:47:52.119437 25422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9bc9b5c7d043c9a5b2d36105b1414c, State: Initialized, Role: FOLLOWER
I20260621 02:47:52.119989 25422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39557 } }
I20260621 02:47:52.120195 25422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:47:52.120287 25422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:47:52.120412 25422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:47:52.122437 25422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39557 } }
I20260621 02:47:52.122877 25422 leader_election.cc:302] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a9bc9b5c7d043c9a5b2d36105b1414c; no voters:
I20260621 02:47:52.123581 25422 leader_election.cc:288] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:47:52.123697 25427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:47:52.124588 25427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 1 LEADER]: Becoming Leader. State: Replica: 6a9bc9b5c7d043c9a5b2d36105b1414c, State: Running, Role: LEADER
I20260621 02:47:52.125118 25427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39557 } }
I20260621 02:47:52.125699 25422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:47:52.128129 25429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a9bc9b5c7d043c9a5b2d36105b1414c. Latest consensus state: current_term: 1 leader_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39557 } } }
I20260621 02:47:52.128084 25428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39557 } } }
I20260621 02:47:52.128546 25429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [sys.catalog]: This master's current role is: LEADER
I20260621 02:47:52.128574 25428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [sys.catalog]: This master's current role is: LEADER
I20260621 02:47:52.130537 25435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:47:52.136909 25435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:47:52.145282 25435 catalog_manager.cc:1347] Generated new cluster ID: 53b6fe39a5fe42dfa6faf338252f6d54
I20260621 02:47:52.145438 25435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:47:52.157554 25435 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:47:52.158854 25435 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:47:52.167228 25435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c: Generated new TSK 0
I20260621 02:47:52.168092 25435 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
I20260621 02:47:52.230635 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:39557
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:47:52.561048 25448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:52.561394 25448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:52.561584 25448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:52.569224 25448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:52.569406 25448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:52.569589 25448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:47:52.579655 25448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:39557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:52.580973 25448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:52.582316 25448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:52.595324 25454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:52.595440 25453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:52.595445 25456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:52.596092 25448 server_base.cc:1034] running on GCE node
I20260621 02:47:52.596715 25448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:52.597621 25448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:52.598882 25448 hybrid_clock.cc:648] HybridClock initialized: now 1782010072598816 us; error 55 us; skew 500 ppm
I20260621 02:47:52.599225 25448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:52 dist-test-slave-hvgn krb5kdc[25342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010072, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:52.602501 25448 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:47:52.603986 25448 webserver.cc:460] Webserver started at http://127.9.166.1:34801/ using document root <none> and password file <none>
I20260621 02:47:52.604692 25448 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:52.604838 25448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:52.605150 25448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:47:52.608058 25448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "cc8e3c5abc7e4551ae9848742981a524"
format_stamp: "Formatted at 2026-06-21 02:47:52 on dist-test-slave-hvgn"
server_key: "908a1a3f5bb68722696ccc7526db3700"
server_key_iv: "1d2e95038339e8fd93e7240600292709"
server_key_version: "encryptionkey@0"
I20260621 02:47:52.608806 25448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "cc8e3c5abc7e4551ae9848742981a524"
format_stamp: "Formatted at 2026-06-21 02:47:52 on dist-test-slave-hvgn"
server_key: "908a1a3f5bb68722696ccc7526db3700"
server_key_iv: "1d2e95038339e8fd93e7240600292709"
server_key_version: "encryptionkey@0"
I20260621 02:47:52.613852 25448 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:47:52.617305 25463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:52.618669 25448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260621 02:47:52.618853 25448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "cc8e3c5abc7e4551ae9848742981a524"
format_stamp: "Formatted at 2026-06-21 02:47:52 on dist-test-slave-hvgn"
server_key: "908a1a3f5bb68722696ccc7526db3700"
server_key_iv: "1d2e95038339e8fd93e7240600292709"
server_key_version: "encryptionkey@0"
I20260621 02:47:52.619052 25448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:52.641223 25448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:52.645646 25448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:52.645953 25448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:52.646960 25448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:47:52.648533 25448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:47:52.648615 25448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:52.648795 25448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:47:52.648852 25448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:52.680660 25448 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:36415
I20260621 02:47:52.680706 25575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:36415 every 8 connection(s)
I20260621 02:47:52.682026 25448 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:47:52.686316 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 25448
I20260621 02:47:52.686530 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:47:52.687153 9880 external_mini_cluster.cc:1428] Setting key baa03015719cad084346e65f0cf11d2a
Jun 21 02:47:52 dist-test-slave-hvgn krb5kdc[25342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010072, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:52.703228 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:52.684532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:60897 (local address 127.9.166.62:39557)
0621 02:47:52.684994 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:52.684999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:52.685849 (+ 850us) server_negotiation.cc:407] Connection header received
0621 02:47:52.687871 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:52.687876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:52.687975 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:52.688106 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:52.689963 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:52.691172 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:52.692753 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:52.693484 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:52.693633 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:52.697256 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:52.697261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:52.697264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:52.697296 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:52.699916 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:52.700812 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:52.700816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:52.700818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:52.700947 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:52.701459 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:52.701462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:52.701464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:52.701717 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:52.701835 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:52.702763 (+ 928us) server_negotiation.cc:299] Negotiation successful
0621 02:47:52.702965 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":247,"threads_started":1}
I20260621 02:47:52.704748 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:52.685009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.1:60897)
0621 02:47:52.685692 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:52.685730 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:52.687572 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:52.688359 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:52.688369 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:52.689135 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:52.689765 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:52.689783 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:52.691338 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:52.691342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:52.692606 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:52.692617 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:52.693692 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:52.693697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:52.693903 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:52.694593 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:52.694618 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:52.697012 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:52.700052 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:52.700063 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:52.700083 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:52.700696 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:52.701089 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:52.701096 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:52.701102 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:52.701341 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:52.701842 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:52.701852 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:52.702302 (+ 450us) client_negotiation.cc:769] Sending connection context
0621 02:47:52.702620 (+ 318us) client_negotiation.cc:240] Negotiation successful
0621 02:47:52.703203 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":262,"threads_started":1}
I20260621 02:47:52.706450 25576 heartbeater.cc:347] Connected to a master server at 127.9.166.62:39557
I20260621 02:47:52.706807 25576 heartbeater.cc:464] Registering TS with master...
I20260621 02:47:52.707736 25576 heartbeater.cc:511] Master 127.9.166.62:39557 requested a full tablet report, sending...
I20260621 02:47:52.710143 25387 ts_manager.cc:194] Registered new tserver with Master: cc8e3c5abc7e4551ae9848742981a524 (127.9.166.1:36415)
I20260621 02:47:52.711704 25387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:60897
WARNING: no policy specified for kudu/127.9.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
I20260621 02:47:52.740778 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:39557
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:47:53.060813 25582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:53.061148 25582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:53.061324 25582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:53.068442 25582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:53.068598 25582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:53.068789 25582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:47:53.078361 25582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:39557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:53.079669 25582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:53.081004 25582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:53.095280 25588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:53.095243 25587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:53.096199 25590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:53.097126 25582 server_base.cc:1034] running on GCE node
I20260621 02:47:53.097736 25582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:53.098800 25582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:53.100076 25582 hybrid_clock.cc:648] HybridClock initialized: now 1782010073100010 us; error 55 us; skew 500 ppm
I20260621 02:47:53.100414 25582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:53 dist-test-slave-hvgn krb5kdc[25342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010073, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:53.103694 25582 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:47:53.105229 25582 webserver.cc:460] Webserver started at http://127.9.166.2:39615/ using document root <none> and password file <none>
I20260621 02:47:53.105942 25582 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:53.106088 25582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:53.106402 25582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:47:53.109391 25582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "7eb4fd2059544a96b0498bfd25216fb9"
format_stamp: "Formatted at 2026-06-21 02:47:53 on dist-test-slave-hvgn"
server_key: "d8040508739196c6411f8ea82e840ee6"
server_key_iv: "bd1b623573f43f7f4146fa117fc2f2eb"
server_key_version: "encryptionkey@0"
I20260621 02:47:53.110181 25582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "7eb4fd2059544a96b0498bfd25216fb9"
format_stamp: "Formatted at 2026-06-21 02:47:53 on dist-test-slave-hvgn"
server_key: "d8040508739196c6411f8ea82e840ee6"
server_key_iv: "bd1b623573f43f7f4146fa117fc2f2eb"
server_key_version: "encryptionkey@0"
I20260621 02:47:53.115525 25582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260621 02:47:53.118942 25597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:53.120352 25582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:47:53.120530 25582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "7eb4fd2059544a96b0498bfd25216fb9"
format_stamp: "Formatted at 2026-06-21 02:47:53 on dist-test-slave-hvgn"
server_key: "d8040508739196c6411f8ea82e840ee6"
server_key_iv: "bd1b623573f43f7f4146fa117fc2f2eb"
server_key_version: "encryptionkey@0"
I20260621 02:47:53.120733 25582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:53.135831 25582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:53.139809 25582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:53.140091 25582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:53.141011 25582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:47:53.142334 25582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:47:53.142432 25582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:53.142585 25582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:47:53.142659 25582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:53.176491 25582 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:34487
I20260621 02:47:53.176537 25709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:34487 every 8 connection(s)
I20260621 02:47:53.177804 25582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:47:53.183524 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 25582
I20260621 02:47:53.183727 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:47:53.184232 9880 external_mini_cluster.cc:1428] Setting key f22e2f2259bbbcec6b35a48204ae24cc
Jun 21 02:47:53 dist-test-slave-hvgn krb5kdc[25342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010073, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:53.197911 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:53.180445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:42209 (local address 127.9.166.62:39557)
0621 02:47:53.180621 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:53.180625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:53.181772 (+ 1147us) server_negotiation.cc:407] Connection header received
0621 02:47:53.183697 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:53.183703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:53.183827 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:53.183957 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:53.185642 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.186459 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.188048 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.188466 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.188605 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:53.192061 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:53.192066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:53.192069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:53.192101 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:53.194724 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:53.195660 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:53.195664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:53.195666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:53.195797 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:53.196280 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:53.196283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:53.196285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:53.196548 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:53.196658 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:53.197514 (+ 856us) server_negotiation.cc:299] Negotiation successful
0621 02:47:53.197680 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:47:53.199555 25712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:53.180831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.2:42209)
0621 02:47:53.181586 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:53.181635 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:53.183360 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:53.184193 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:53.184202 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:53.184787 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:53.185481 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:53.185498 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.186603 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.186613 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:53.187896 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:53.187908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.188700 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.188707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:53.188927 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:53.189641 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:53.189668 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:53.191837 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:53.194862 (+ 3025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:53.194874 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:53.194892 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:53.195540 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:53.195912 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:53.195919 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:53.195926 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:53.196169 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:53.196680 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:53.196691 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:53.197125 (+ 434us) client_negotiation.cc:769] Sending connection context
0621 02:47:53.197431 (+ 306us) client_negotiation.cc:240] Negotiation successful
0621 02:47:53.197977 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":281,"threads_started":1}
I20260621 02:47:53.200977 25710 heartbeater.cc:347] Connected to a master server at 127.9.166.62:39557
I20260621 02:47:53.201295 25710 heartbeater.cc:464] Registering TS with master...
I20260621 02:47:53.202172 25710 heartbeater.cc:511] Master 127.9.166.62:39557 requested a full tablet report, sending...
I20260621 02:47:53.203745 25387 ts_manager.cc:194] Registered new tserver with Master: 7eb4fd2059544a96b0498bfd25216fb9 (127.9.166.2:34487)
I20260621 02:47:53.204577 25387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:42209
WARNING: no policy specified for kudu/127.9.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
I20260621 02:47:53.238448 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:39557
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:47:53.561653 25715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:47:53.561972 25715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:47:53.562137 25715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:47:53.569439 25715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:47:53.569590 25715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:47:53.569774 25715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:47:53.580005 25715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:39557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:47:53.581295 25715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:47:53.582628 25715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:47:53.595338 25720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:53.595378 25721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:47:53.595422 25723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:47:53.596948 25715 server_base.cc:1034] running on GCE node
I20260621 02:47:53.597534 25715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:47:53.598408 25715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:47:53.599592 25715 hybrid_clock.cc:648] HybridClock initialized: now 1782010073599528 us; error 49 us; skew 500 ppm
I20260621 02:47:53.599906 25715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:47:53 dist-test-slave-hvgn krb5kdc[25342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010073, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:47:53.608168 25715 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:47:53.609700 25715 webserver.cc:460] Webserver started at http://127.9.166.3:39907/ using document root <none> and password file <none>
I20260621 02:47:53.610406 25715 fs_manager.cc:359] Metadata directory not provided
I20260621 02:47:53.610530 25715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:47:53.610862 25715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:47:53.613798 25715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "b14ec0114d2f4eaab780f5fc0425fbe9"
format_stamp: "Formatted at 2026-06-21 02:47:53 on dist-test-slave-hvgn"
server_key: "7fa7cc0f7fcd51d94c1ac874808cd8f3"
server_key_iv: "6da50083227690b0a8a2c65f84d0cf32"
server_key_version: "encryptionkey@0"
I20260621 02:47:53.614552 25715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "b14ec0114d2f4eaab780f5fc0425fbe9"
format_stamp: "Formatted at 2026-06-21 02:47:53 on dist-test-slave-hvgn"
server_key: "7fa7cc0f7fcd51d94c1ac874808cd8f3"
server_key_iv: "6da50083227690b0a8a2c65f84d0cf32"
server_key_version: "encryptionkey@0"
I20260621 02:47:53.619807 25715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260621 02:47:53.623306 25730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:53.624787 25715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:47:53.624966 25715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "b14ec0114d2f4eaab780f5fc0425fbe9"
format_stamp: "Formatted at 2026-06-21 02:47:53 on dist-test-slave-hvgn"
server_key: "7fa7cc0f7fcd51d94c1ac874808cd8f3"
server_key_iv: "6da50083227690b0a8a2c65f84d0cf32"
server_key_version: "encryptionkey@0"
I20260621 02:47:53.625156 25715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:47:53.636544 25715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:47:53.640522 25715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:47:53.640753 25715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:47:53.641602 25715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:47:53.642834 25715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:47:53.642904 25715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:53.642984 25715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:47:53.643028 25715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:47:53.676389 25715 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:40409
I20260621 02:47:53.676440 25842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:40409 every 8 connection(s)
I20260621 02:47:53.677826 25715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:47:53.680546 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 25715
I20260621 02:47:53.680735 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:47:53.681337 9880 external_mini_cluster.cc:1428] Setting key 558de62555e77bf36630e25eaaa6f2d9
Jun 21 02:47:53 dist-test-slave-hvgn krb5kdc[25342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010073, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:47:53.697495 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:53.680487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:42283 (local address 127.9.166.62:39557)
0621 02:47:53.681524 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:53.681530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:53.681991 (+ 461us) server_negotiation.cc:407] Connection header received
0621 02:47:53.683987 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:53.683993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:53.684098 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:53.684212 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:53.685746 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.686675 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.688254 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.688644 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.688735 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:53.692124 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:53.692129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:53.692132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:53.692161 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:53.694752 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:53.695518 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:53.695522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:53.695524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:53.695645 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:53.696004 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:53.696007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:53.696009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:53.696244 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:53.696358 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:53.696973 (+ 615us) server_negotiation.cc:299] Negotiation successful
0621 02:47:53.697164 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":861}
I20260621 02:47:53.698544 25845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:53.681051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.3:42283)
0621 02:47:53.681817 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:53.681881 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:53.683709 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:53.684463 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:53.684472 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:53.685072 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:47:53.685586 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:53.685601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.686821 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.686825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:53.688120 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:53.688136 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.688847 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.688855 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:53.689136 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:53.689835 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:47:53.689856 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:47:53.691946 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:47:53.694941 (+ 2995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:53.694947 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:53.694957 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:53.695369 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:53.695732 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:47:53.695735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:47:53.695737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:47:53.695904 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:47:53.696376 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:47:53.696381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:47:53.696640 (+ 259us) client_negotiation.cc:769] Sending connection context
0621 02:47:53.696889 (+ 249us) client_negotiation.cc:240] Negotiation successful
0621 02:47:53.697349 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":417,"threads_started":1}
I20260621 02:47:53.699868 25843 heartbeater.cc:347] Connected to a master server at 127.9.166.62:39557
I20260621 02:47:53.700163 25843 heartbeater.cc:464] Registering TS with master...
I20260621 02:47:53.700836 25843 heartbeater.cc:511] Master 127.9.166.62:39557 requested a full tablet report, sending...
I20260621 02:47:53.702157 25387 ts_manager.cc:194] Registered new tserver with Master: b14ec0114d2f4eaab780f5fc0425fbe9 (127.9.166.3:40409)
I20260621 02:47:53.702879 25387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:42283
I20260621 02:47:53.711138 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:47:53.714920 25576 heartbeater.cc:503] Master 127.9.166.62:39557 was elected leader, sending a full tablet report...
I20260621 02:47:53.728380 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:53.715614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59872 (local address 127.9.166.62:39557)
0621 02:47:53.715826 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:53.715833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:53.716182 (+ 349us) server_negotiation.cc:407] Connection header received
0621 02:47:53.716385 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:53.716390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:53.716462 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:53.716554 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:53.717977 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.718682 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.720000 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.720414 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.720510 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:53.722887 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:53.722892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:53.722895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:53.722924 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:53.725842 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:53.726525 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:53.726529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:53.726531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:53.726690 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:53.727023 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:53.727026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:53.727028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:53.727453 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:53.727643 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:53.727945 (+ 302us) server_negotiation.cc:299] Negotiation successful
0621 02:47:53.728095 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 02:47:53.732380 25387 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:47:53.739194 25387 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:47:53.762036 25855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:53.751046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:36415 (local address 127.0.0.1:47234)
0621 02:47:53.753218 (+ 2172us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:53.753231 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:53.753309 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:53.754398 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:53.754406 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:53.754436 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:53.754883 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:53.754899 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.756507 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.756515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:53.759141 (+ 2626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:53.759155 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.760756 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.760764 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:53.761027 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:53.761097 (+ 70us) client_negotiation.cc:769] Sending connection context
0621 02:47:53.761235 (+ 138us) client_negotiation.cc:240] Negotiation successful
0621 02:47:53.761528 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2066,"mutex_wait_us":69,"thread_start_us":1233,"threads_started":1}
I20260621 02:47:53.762707 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:53.751444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.9.166.1:36415)
0621 02:47:53.753797 (+ 2353us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:53.753804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:53.753824 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:47:53.753904 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:53.753909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:53.754157 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:53.755119 (+ 962us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:53.755323 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.756377 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.759316 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.760537 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.760675 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:53.760921 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:53.761300 (+ 379us) server_negotiation.cc:299] Negotiation successful
0621 02:47:53.761465 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2215,"thread_start_us":2640,"threads_started":1}
I20260621 02:47:53.764494 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:53.750840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:34487 (local address 127.0.0.1:34028)
0621 02:47:53.752561 (+ 1721us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:53.752612 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:53.752784 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:53.757492 (+ 4708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:53.757499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:53.757529 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:53.758009 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:53.758023 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.760065 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.760071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:53.762050 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:53.762064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.763727 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.763733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:53.763917 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:53.763940 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 02:47:53.764043 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 02:47:53.764164 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1481,"thread_start_us":2172,"threads_started":1}
I20260621 02:47:53.764562 25857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:53.751037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34028 (local address 127.9.166.2:34487)
0621 02:47:53.756918 (+ 5881us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:53.756926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:53.756952 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:47:53.757038 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:53.757045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:53.757276 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:53.758214 (+ 938us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:53.758411 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.759929 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.762234 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.763527 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.763670 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:53.763862 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:53.764086 (+ 224us) server_negotiation.cc:299] Negotiation successful
0621 02:47:53.764239 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5703,"thread_start_us":5593,"threads_started":1}
I20260621 02:47:53.765923 25859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:53.753491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:40409 (local address 127.0.0.1:57506)
0621 02:47:53.757866 (+ 4375us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:53.757888 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:53.758022 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:53.758894 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:53.758897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:53.758915 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:53.759219 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:53.759253 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.760848 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.760852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:53.763360 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:53.763373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.765134 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.765140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:53.765298 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:53.765329 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 02:47:53.765422 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 02:47:53.765545 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4192,"mutex_wait_us":21,"thread_start_us":4031,"threads_started":1}
I20260621 02:47:53.765951 25860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:53.753905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57506 (local address 127.9.166.3:40409)
0621 02:47:53.757343 (+ 3438us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:53.757354 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:53.758147 (+ 793us) server_negotiation.cc:407] Connection header received
0621 02:47:53.758246 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:53.758253 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:53.758486 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:53.758660 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:53.759452 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.760648 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.763566 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:53.764938 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:53.765065 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:53.765286 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:53.765474 (+ 188us) server_negotiation.cc:299] Negotiation successful
0621 02:47:53.765632 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3226,"thread_start_us":3079,"threads_started":1}
I20260621 02:47:53.766621 25511 tablet_service.cc:1467] Processing CreateTablet for tablet befa2eab885c4af998c137dc30b23c7b (DEFAULT_TABLE table=test-table [id=d5fdd995a5814687bdd1b03a5aed825f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:47:53.767972 25511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet befa2eab885c4af998c137dc30b23c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:47:53.768110 25645 tablet_service.cc:1467] Processing CreateTablet for tablet befa2eab885c4af998c137dc30b23c7b (DEFAULT_TABLE table=test-table [id=d5fdd995a5814687bdd1b03a5aed825f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:47:53.769136 25645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet befa2eab885c4af998c137dc30b23c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:47:53.774899 25778 tablet_service.cc:1467] Processing CreateTablet for tablet befa2eab885c4af998c137dc30b23c7b (DEFAULT_TABLE table=test-table [id=d5fdd995a5814687bdd1b03a5aed825f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:47:53.776252 25778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet befa2eab885c4af998c137dc30b23c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:47:53.779279 25861 tablet_bootstrap.cc:492] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524: Bootstrap starting.
I20260621 02:47:53.782078 25862 tablet_bootstrap.cc:492] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9: Bootstrap starting.
I20260621 02:47:53.783725 25861 tablet_bootstrap.cc:654] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524: Neither blocks nor log segments found. Creating new log.
I20260621 02:47:53.784973 25861 log.cc:824] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:53.785452 25862 tablet_bootstrap.cc:654] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9: Neither blocks nor log segments found. Creating new log.
I20260621 02:47:53.786612 25862 log.cc:824] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:53.787981 25861 tablet_bootstrap.cc:492] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524: No bootstrap required, opened a new log
I20260621 02:47:53.788244 25861 ts_tablet_manager.cc:1397] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260621 02:47:53.789594 25862 tablet_bootstrap.cc:492] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9: No bootstrap required, opened a new log
I20260621 02:47:53.789589 25865 tablet_bootstrap.cc:492] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9: Bootstrap starting.
I20260621 02:47:53.789949 25862 ts_tablet_manager.cc:1397] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260621 02:47:53.792542 25865 tablet_bootstrap.cc:654] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9: Neither blocks nor log segments found. Creating new log.
I20260621 02:47:53.793452 25865 log.cc:824] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9: Log is configured to *not* fsync() on all Append() calls
I20260621 02:47:53.795781 25865 tablet_bootstrap.cc:492] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9: No bootstrap required, opened a new log
I20260621 02:47:53.796028 25865 ts_tablet_manager.cc:1397] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260621 02:47:53.800637 25861 raft_consensus.cc:348] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 } } peers { permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 } } peers { permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 } }
I20260621 02:47:53.800637 25862 raft_consensus.cc:348] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 } } peers { permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 } } peers { permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 } }
I20260621 02:47:53.801066 25862 raft_consensus.cc:374] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:47:53.801198 25861 raft_consensus.cc:374] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:47:53.801201 25862 raft_consensus.cc:729] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4fd2059544a96b0498bfd25216fb9, State: Initialized, Role: FOLLOWER
I20260621 02:47:53.801311 25861 raft_consensus.cc:729] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8e3c5abc7e4551ae9848742981a524, State: Initialized, Role: FOLLOWER
I20260621 02:47:53.801925 25861 consensus_queue.cc:260] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 } } peers { permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 } } peers { permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 } }
I20260621 02:47:53.801924 25862 consensus_queue.cc:260] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 } } peers { permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 } } peers { permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 } }
I20260621 02:47:53.803526 25710 heartbeater.cc:503] Master 127.9.166.62:39557 was elected leader, sending a full tablet report...
I20260621 02:47:53.803930 25861 ts_tablet_manager.cc:1428] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260621 02:47:53.803797 25865 raft_consensus.cc:348] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 } } peers { permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 } } peers { permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 } }
I20260621 02:47:53.804260 25865 raft_consensus.cc:374] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:47:53.804411 25865 raft_consensus.cc:729] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b14ec0114d2f4eaab780f5fc0425fbe9, State: Initialized, Role: FOLLOWER
I20260621 02:47:53.805056 25862 ts_tablet_manager.cc:1428] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260621 02:47:53.805145 25865 consensus_queue.cc:260] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 } } peers { permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 } } peers { permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 } }
I20260621 02:47:53.806200 25843 heartbeater.cc:503] Master 127.9.166.62:39557 was elected leader, sending a full tablet report...
I20260621 02:47:53.806839 25865 ts_tablet_manager.cc:1428] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
W20260621 02:47:53.929481 25844 tablet.cc:2369] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:47:53.929845 25711 tablet.cc:2369] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:47:53.934583 25577 tablet.cc:2369] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:47:54.122895 25868 raft_consensus.cc:482] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:47:54.123212 25868 raft_consensus.cc:504] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 } } peers { permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 } } peers { permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 } }
I20260621 02:47:54.125075 25868 leader_election.cc:288] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc8e3c5abc7e4551ae9848742981a524 (127.9.166.1:36415), b14ec0114d2f4eaab780f5fc0425fbe9 (127.9.166.3:40409)
I20260621 02:47:54.132190 25860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:54.125680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:48137 (local address 127.9.166.3:40409)
0621 02:47:54.125954 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:54.125963 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:54.126545 (+ 582us) server_negotiation.cc:407] Connection header received
0621 02:47:54.126815 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:54.126819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:54.126880 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:54.126972 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:54.127754 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:54.128490 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:54.130454 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:54.131216 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:54.131367 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:54.131495 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:54.131796 (+ 301us) server_negotiation.cc:299] Negotiation successful
0621 02:47:54.131920 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260621 02:47:54.132334 25871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:54.125399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:40409 (local address 127.9.166.2:48137)
0621 02:47:54.126426 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:54.126445 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:54.126651 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:54.127040 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:54.127045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:54.127120 (+ 75us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:54.127605 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:54.127615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:54.128622 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:54.128629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:54.130293 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:54.130305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:54.131461 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:54.131467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:54.131608 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:54.131634 (+ 26us) client_negotiation.cc:769] Sending connection context
0621 02:47:54.131770 (+ 136us) client_negotiation.cc:240] Negotiation successful
0621 02:47:54.131894 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"mutex_wait_us":1,"spinlock_wait_cycles":25344,"thread_start_us":802,"threads_started":1}
I20260621 02:47:54.132419 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:54.125455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:34151 (local address 127.9.166.1:36415)
0621 02:47:54.125724 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:54.125734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:54.126633 (+ 899us) server_negotiation.cc:407] Connection header received
0621 02:47:54.126834 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:54.126842 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:54.126926 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:54.127017 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:47:54.127717 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:54.128533 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:54.130610 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:54.131426 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:54.131584 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:54.131685 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:54.132024 (+ 339us) server_negotiation.cc:299] Negotiation successful
0621 02:47:54.132163 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260621 02:47:54.132586 25870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:54.125438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:36415 (local address 127.9.166.2:34151)
0621 02:47:54.126524 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0621 02:47:54.126538 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 02:47:54.126682 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:47:54.127044 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:47:54.127048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:47:54.127078 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:47:54.127563 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:54.127573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:54.128689 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:54.128694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:54.130462 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:47:54.130474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:47:54.131640 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:47:54.131646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:47:54.131827 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:54.131849 (+ 22us) client_negotiation.cc:769] Sending connection context
0621 02:47:54.132007 (+ 158us) client_negotiation.cc:240] Negotiation successful
0621 02:47:54.132129 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"mutex_wait_us":23,"thread_start_us":605,"threads_started":1}
I20260621 02:47:54.133522 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "befa2eab885c4af998c137dc30b23c7b" candidate_uuid: "7eb4fd2059544a96b0498bfd25216fb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" is_pre_election: true
I20260621 02:47:54.133610 25531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "befa2eab885c4af998c137dc30b23c7b" candidate_uuid: "7eb4fd2059544a96b0498bfd25216fb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8e3c5abc7e4551ae9848742981a524" is_pre_election: true
I20260621 02:47:54.133992 25798 raft_consensus.cc:2454] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4fd2059544a96b0498bfd25216fb9 in term 0.
I20260621 02:47:54.134133 25531 raft_consensus.cc:2454] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4fd2059544a96b0498bfd25216fb9 in term 0.
I20260621 02:47:54.134640 25601 leader_election.cc:302] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4fd2059544a96b0498bfd25216fb9, b14ec0114d2f4eaab780f5fc0425fbe9; no voters:
I20260621 02:47:54.135020 25868 raft_consensus.cc:2781] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:47:54.135164 25868 raft_consensus.cc:482] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:47:54.135264 25868 raft_consensus.cc:3037] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:47:54.137509 25868 raft_consensus.cc:504] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 } } peers { permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 } } peers { permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 } }
I20260621 02:47:54.138141 25868 leader_election.cc:288] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [CANDIDATE]: Term 1 election: Requested vote from peers cc8e3c5abc7e4551ae9848742981a524 (127.9.166.1:36415), b14ec0114d2f4eaab780f5fc0425fbe9 (127.9.166.3:40409)
I20260621 02:47:54.138499 25531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "befa2eab885c4af998c137dc30b23c7b" candidate_uuid: "7eb4fd2059544a96b0498bfd25216fb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8e3c5abc7e4551ae9848742981a524"
I20260621 02:47:54.138618 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "befa2eab885c4af998c137dc30b23c7b" candidate_uuid: "7eb4fd2059544a96b0498bfd25216fb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9"
I20260621 02:47:54.138707 25531 raft_consensus.cc:3037] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:47:54.138821 25798 raft_consensus.cc:3037] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:47:54.141870 25531 raft_consensus.cc:2454] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4fd2059544a96b0498bfd25216fb9 in term 1.
I20260621 02:47:54.141870 25798 raft_consensus.cc:2454] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4fd2059544a96b0498bfd25216fb9 in term 1.
I20260621 02:47:54.142244 25598 leader_election.cc:302] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4fd2059544a96b0498bfd25216fb9, cc8e3c5abc7e4551ae9848742981a524; no voters:
I20260621 02:47:54.142518 25868 raft_consensus.cc:2781] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:47:54.142949 25868 raft_consensus.cc:686] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4fd2059544a96b0498bfd25216fb9, State: Running, Role: LEADER
I20260621 02:47:54.143442 25868 consensus_queue.cc:237] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 } } peers { permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 } } peers { permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 } }
I20260621 02:47:54.148030 25387 catalog_manager.cc:5495] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4fd2059544a96b0498bfd25216fb9 (127.9.166.2). New cstate: current_term: 1 leader_uuid: "7eb4fd2059544a96b0498bfd25216fb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 } health_report { overall_health: UNKNOWN } } }
I20260621 02:47:54.214454 25857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:54.209218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34044 (local address 127.9.166.2:34487)
0621 02:47:54.209428 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:54.209433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:54.209450 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:47:54.209607 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:54.209612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:54.209687 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:54.209811 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:47:54.210135 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:54.210877 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:54.212250 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:54.212702 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:54.212801 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:54.213304 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:47:54.213891 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:47:54.214058 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:54.214097 (+ 39us) server_negotiation.cc:299] Negotiation successful
0621 02:47:54.214158 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 02:47:54.222263 25798 raft_consensus.cc:1261] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4fd2059544a96b0498bfd25216fb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:47:54.222381 25531 raft_consensus.cc:1261] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4fd2059544a96b0498bfd25216fb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:47:54.223325 25872 consensus_queue.cc:1035] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:47:54.223706 25868 consensus_queue.cc:1035] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:47:54.236171 25876 mvcc.cc:205] Tried to move back new op lower bound from 7299113264005480448 to 7299113263695036416. Current Snapshot: MvccSnapshot[applied={T|T < 7299113264005480448}]
I20260621 02:47:54.236654 25875 mvcc.cc:205] Tried to move back new op lower bound from 7299113264005480448 to 7299113263695036416. Current Snapshot: MvccSnapshot[applied={T|T < 7299113264005480448}]
I20260621 02:47:54.249840 25386 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:47:54.250115 25386 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:47:54.253391 25386 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c: Sending DeleteTablet for 3 replicas of tablet befa2eab885c4af998c137dc30b23c7b
I20260621 02:47:54.254575 25645 tablet_service.cc:1514] Processing DeleteTablet for tablet befa2eab885c4af998c137dc30b23c7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:47:54 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:34028
I20260621 02:47:54.254631 25511 tablet_service.cc:1514] Processing DeleteTablet for tablet befa2eab885c4af998c137dc30b23c7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:47:54 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:47234
I20260621 02:47:54.254721 25778 tablet_service.cc:1514] Processing DeleteTablet for tablet befa2eab885c4af998c137dc30b23c7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:47:54 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:57506
I20260621 02:47:54.255514 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 25448
I20260621 02:47:54.267453 25886 tablet_replica.cc:330] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9: stopping tablet replica
I20260621 02:47:54.268108 25886 raft_consensus.cc:2229] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:47:54.268749 25886 raft_consensus.cc:2258] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:47:54.271234 25886 ts_tablet_manager.cc:1905] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:47:54.278316 25886 ts_tablet_manager.cc:1918] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:47:54.278546 25886 log.cc:1196] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/befa2eab885c4af998c137dc30b23c7b
I20260621 02:47:54.279151 25886 ts_tablet_manager.cc:1939] T befa2eab885c4af998c137dc30b23c7b P b14ec0114d2f4eaab780f5fc0425fbe9: Deleting consensus metadata
I20260621 02:47:54.281188 25374 catalog_manager.cc:4841] TS b14ec0114d2f4eaab780f5fc0425fbe9 (127.9.166.3:40409): tablet befa2eab885c4af998c137dc30b23c7b (table test-table [id=d5fdd995a5814687bdd1b03a5aed825f]) successfully deleted
I20260621 02:47:54.282567 25884 tablet_replica.cc:330] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9: stopping tablet replica
I20260621 02:47:54.283026 25884 raft_consensus.cc:2229] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:47:54.283696 25884 raft_consensus.cc:2258] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:47:54.284479 25885 tablet_replica.cc:330] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524: stopping tablet replica
I20260621 02:47:54.284898 25885 raft_consensus.cc:2229] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:47:54.285385 25885 raft_consensus.cc:2258] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:47:54.286193 25884 ts_tablet_manager.cc:1905] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:47:54.287333 25885 ts_tablet_manager.cc:1905] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 21 02:47:54 dist-test-slave-hvgn krb5kdc[25342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:47:54.294276 25885 ts_tablet_manager.cc:1918] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:47:54.294489 25885 log.cc:1196] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/befa2eab885c4af998c137dc30b23c7b
I20260621 02:47:54.294796 25884 ts_tablet_manager.cc:1918] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:47:54.295078 25885 ts_tablet_manager.cc:1939] T befa2eab885c4af998c137dc30b23c7b P cc8e3c5abc7e4551ae9848742981a524: Deleting consensus metadata
I20260621 02:47:54.295008 25884 log.cc:1196] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/befa2eab885c4af998c137dc30b23c7b
I20260621 02:47:54.295967 25884 ts_tablet_manager.cc:1939] T befa2eab885c4af998c137dc30b23c7b P 7eb4fd2059544a96b0498bfd25216fb9: Deleting consensus metadata
I20260621 02:47:54.296950 25371 catalog_manager.cc:4841] TS cc8e3c5abc7e4551ae9848742981a524 (127.9.166.1:36415): tablet befa2eab885c4af998c137dc30b23c7b (table test-table [id=d5fdd995a5814687bdd1b03a5aed825f]) successfully deleted
I20260621 02:47:54.297555 25374 catalog_manager.cc:4841] TS 7eb4fd2059544a96b0498bfd25216fb9 (127.9.166.2:34487): tablet befa2eab885c4af998c137dc30b23c7b (table test-table [id=d5fdd995a5814687bdd1b03a5aed825f]) successfully deleted
I20260621 02:47:54.300087 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:54.256870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47240 (local address 127.9.166.1:36415)
0621 02:47:54.257085 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:54.257095 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:54.280187 (+ 23092us) server_negotiation.cc:407] Connection header received
0621 02:47:54.280855 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:54.280864 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:54.281002 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:54.281096 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:54.283179 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:54.284176 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:54.286249 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:54.286897 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:54.286993 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:54.293036 (+ 6043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:54.293048 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:54.293065 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:54.293116 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:54.297309 (+ 4193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:54.298145 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:54.298152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:54.298154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:54.298319 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:54.298728 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:54.298731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:54.298733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:54.299045 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:54.299294 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:54.299646 (+ 352us) server_negotiation.cc:299] Negotiation successful
0621 02:47:54.299806 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:47:54.300894 25571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:54.392102 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 25448
I20260621 02:47:54.409803 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 25582
Jun 21 02:47:54 dist-test-slave-hvgn krb5kdc[25342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.2@KRBTEST.COM
I20260621 02:47:54.427947 25857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:54.410669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34060 (local address 127.9.166.2:34487)
0621 02:47:54.410856 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:54.410867 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:54.410892 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:47:54.411077 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:54.411083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:54.411175 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:54.411294 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:54.412705 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:54.413952 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:54.415451 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:54.416102 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:54.416234 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:54.420901 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:54.420919 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:54.420939 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:54.420986 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:54.424935 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:54.425724 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:54.425735 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:54.425739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:54.425896 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:54.426220 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:54.426225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:54.426229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:54.426717 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:54.427033 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:54.427314 (+ 281us) server_negotiation.cc:299] Negotiation successful
0621 02:47:54.427557 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260621 02:47:54.428755 25705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:54.544106 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 25582
I20260621 02:47:54.562999 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 25715
Jun 21 02:47:54 dist-test-slave-hvgn krb5kdc[25342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.3@KRBTEST.COM
I20260621 02:47:54.577934 25860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:47:54.563730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57510 (local address 127.9.166.3:40409)
0621 02:47:54.563906 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0621 02:47:54.563911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:47:54.563929 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:47:54.564062 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:47:54.564066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:47:54.564136 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:47:54.564220 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:47:54.565632 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:54.566386 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:54.567948 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:47:54.568348 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:47:54.568445 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:47:54.572818 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:47:54.572830 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:47:54.572846 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:47:54.572894 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:47:54.575586 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:54.576193 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:54.576200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:54.576203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:54.576330 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:47:54.576621 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:47:54.576624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:47:54.576626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:47:54.576966 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:47:54.577173 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0621 02:47:54.577417 (+ 244us) server_negotiation.cc:299] Negotiation successful
0621 02:47:54.577568 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:47:54.578557 25838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:54.667452 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 25715
I20260621 02:47:54.684185 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 25355
I20260621 02:47:54.685272 25417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:47:54.793799 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 25355
I20260621 02:48:14.815591 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:39557
--webserver_interface=127.9.166.62
--webserver_port=43155
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:39557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:48:15.143322 25892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:48:15.143623 25892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:48:15.143782 25892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:48:15.150818 25892 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 02:48:15.150906 25892 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 02:48:15.150944 25892 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 02:48:15.151067 25892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:48:15.151151 25892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:48:15.151316 25892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:48:15.151368 25892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:48:15.151499 25892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:48:15.161041 25892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:39557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:39557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=43155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:48:15.162334 25892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:48:15.163686 25892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:48:15.186244 25901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:15.186268 25898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:15.186308 25899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:48:15.186897 25892 server_base.cc:1034] running on GCE node
I20260621 02:48:15.187860 25892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:48:15.189420 25892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:48:15.190652 25892 hybrid_clock.cc:648] HybridClock initialized: now 1782010095190583 us; error 39 us; skew 500 ppm
I20260621 02:48:15.191042 25892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:48:15 dist-test-slave-hvgn krb5kdc[25342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010095, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:48:15.198457 25892 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:48:15.200088 25892 webserver.cc:460] Webserver started at http://127.9.166.62:43155/ using document root <none> and password file <none>
I20260621 02:48:15.200865 25892 fs_manager.cc:359] Metadata directory not provided
I20260621 02:48:15.201013 25892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:48:15.206874 25892 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260621 02:48:15.209743 25908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:15.211093 25892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 02:48:15.211279 25892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c"
format_stamp: "Formatted at 2026-06-21 02:47:52 on dist-test-slave-hvgn"
server_key: "07184a62abd3b3472b81ba7c8159d7da"
server_key_iv: "0a85f39fd3133200e950f79d31052683"
server_key_version: "encryptionkey@0"
I20260621 02:48:15.212386 25892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:48:15.234373 25892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:48:15.238673 25892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:48:15.238973 25892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:48:15.258088 25892 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:39557
I20260621 02:48:15.258170 25959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:39557 every 8 connection(s)
I20260621 02:48:15.259632 25892 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:48:15.266619 25960 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:48:15.268747 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 25892
I20260621 02:48:15.269827 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:36415
--local_ip_for_outbound_sockets=127.9.166.1
--tserver_master_addrs=127.9.166.62:39557
--webserver_port=34801
--webserver_interface=127.9.166.1
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:48:15.270735 25960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c: Bootstrap starting.
I20260621 02:48:15.292783 25960 log.cc:824] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:48:15.304535 25960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:48:15.305085 25960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c: Bootstrap complete.
I20260621 02:48:15.314512 25960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39557 } }
I20260621 02:48:15.315338 25960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9bc9b5c7d043c9a5b2d36105b1414c, State: Initialized, Role: FOLLOWER
I20260621 02:48:15.316071 25960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39557 } }
I20260621 02:48:15.316337 25960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:48:15.316461 25960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:48:15.316658 25960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 1 FOLLOWER]: Advancing to term 2
I20260621 02:48:15.320683 25960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39557 } }
I20260621 02:48:15.321218 25960 leader_election.cc:302] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a9bc9b5c7d043c9a5b2d36105b1414c; no voters:
I20260621 02:48:15.321882 25960 leader_election.cc:288] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260621 02:48:15.322005 25964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 2 FOLLOWER]: Leader election won for term 2
I20260621 02:48:15.323215 25964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 2 LEADER]: Becoming Leader. State: Replica: 6a9bc9b5c7d043c9a5b2d36105b1414c, State: Running, Role: LEADER
I20260621 02:48:15.323771 25964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39557 } }
I20260621 02:48:15.324316 25960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:48:15.326169 25966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a9bc9b5c7d043c9a5b2d36105b1414c. Latest consensus state: current_term: 2 leader_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39557 } } }
I20260621 02:48:15.326525 25966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [sys.catalog]: This master's current role is: LEADER
I20260621 02:48:15.327152 25965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39557 } } }
I20260621 02:48:15.327503 25965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [sys.catalog]: This master's current role is: LEADER
I20260621 02:48:15.328305 25971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:48:15.351482 25971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:48:15.353600 25971 catalog_manager.cc:1259] Loaded cluster ID: 53b6fe39a5fe42dfa6faf338252f6d54
I20260621 02:48:15.353969 25971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:48:15.358462 25971 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:48:15.360049 25971 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c: Loaded TSK: 0
I20260621 02:48:15.396277 25971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c: Generated new TSK 1
I20260621 02:48:15.396692 25971 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 02:48:15.627588 25962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:48:15.627883 25962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:48:15.628048 25962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:48:15.635262 25962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:48:15.635425 25962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:48:15.635613 25962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:48:15.645123 25962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:36415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=34801
--tserver_master_addrs=127.9.166.62:39557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:48:15.646404 25962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:48:15.647825 25962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:48:15.662803 25987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:15.662765 25988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:15.663884 25990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:48:15.664582 25962 server_base.cc:1034] running on GCE node
I20260621 02:48:15.665190 25962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:48:15.666200 25962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:48:15.667480 25962 hybrid_clock.cc:648] HybridClock initialized: now 1782010095667411 us; error 51 us; skew 500 ppm
I20260621 02:48:15.667806 25962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:48:15 dist-test-slave-hvgn krb5kdc[25342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010095, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:48:15.671214 25962 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:48:15.672937 25962 webserver.cc:460] Webserver started at http://127.9.166.1:34801/ using document root <none> and password file <none>
I20260621 02:48:15.673638 25962 fs_manager.cc:359] Metadata directory not provided
I20260621 02:48:15.673794 25962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:48:15.679184 25962 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:48:15.682044 25997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:15.683527 25962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:48:15.683691 25962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "cc8e3c5abc7e4551ae9848742981a524"
format_stamp: "Formatted at 2026-06-21 02:47:52 on dist-test-slave-hvgn"
server_key: "908a1a3f5bb68722696ccc7526db3700"
server_key_iv: "1d2e95038339e8fd93e7240600292709"
server_key_version: "encryptionkey@0"
I20260621 02:48:15.684646 25962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:48:15.701339 25962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:48:15.705587 25962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:48:15.705869 25962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:48:15.706893 25962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:48:15.708396 25962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:48:15.708532 25962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:15.708663 25962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:48:15.708752 25962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:15.743966 25962 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:36415
I20260621 02:48:15.744051 26109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:36415 every 8 connection(s)
I20260621 02:48:15.745417 25962 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:48:15.746799 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 25962
I20260621 02:48:15.747648 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:34487
--local_ip_for_outbound_sockets=127.9.166.2
--tserver_master_addrs=127.9.166.62:39557
--webserver_port=39615
--webserver_interface=127.9.166.2
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:48:15 dist-test-slave-hvgn krb5kdc[25342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010095, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:48:15.769172 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:15.749163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.1:34561)
0621 02:48:15.749899 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:15.749943 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:48:15.751714 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:48:15.753281 (+ 1567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:48:15.753291 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:48:15.753939 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:48:15.754480 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:15.754498 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:15.756167 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:15.756174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:15.757403 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:15.757415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:15.758456 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:15.758463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:15.758737 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:15.759476 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:48:15.759504 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:48:15.761593 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:48:15.764944 (+ 3351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:48:15.764953 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:48:15.764968 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:48:15.765368 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:48:15.765811 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:48:15.765816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:48:15.765819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:48:15.766011 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:48:15.766711 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:48:15.766719 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:48:15.767052 (+ 333us) client_negotiation.cc:769] Sending connection context
0621 02:48:15.767315 (+ 263us) client_negotiation.cc:240] Negotiation successful
0621 02:48:15.767853 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":308,"threads_started":1}
I20260621 02:48:15.769172 26112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:15.748799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:34561 (local address 127.9.166.62:39557)
0621 02:48:15.749393 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:15.749405 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:15.750080 (+ 675us) server_negotiation.cc:407] Connection header received
0621 02:48:15.752280 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:15.752293 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:15.752708 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:15.753077 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:48:15.754688 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:15.756006 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:15.757565 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:15.758213 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:15.758422 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:15.761867 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:48:15.761874 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:48:15.761888 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:48:15.761923 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:48:15.764787 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:15.765513 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:15.765528 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:15.765534 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:15.765687 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:15.766123 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:15.766129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:15.766132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:15.766572 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:48:15.766784 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:15.767561 (+ 777us) server_negotiation.cc:299] Negotiation successful
0621 02:48:15.767955 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":274,"threads_started":1}
I20260621 02:48:15.772138 26110 heartbeater.cc:347] Connected to a master server at 127.9.166.62:39557
I20260621 02:48:15.772446 26110 heartbeater.cc:464] Registering TS with master...
I20260621 02:48:15.773183 26110 heartbeater.cc:511] Master 127.9.166.62:39557 requested a full tablet report, sending...
I20260621 02:48:15.775226 25925 ts_manager.cc:194] Registered new tserver with Master: cc8e3c5abc7e4551ae9848742981a524 (127.9.166.1:36415)
I20260621 02:48:15.776695 25925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:34561
W20260621 02:48:16.071074 26114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:48:16.071363 26114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:48:16.071547 26114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:48:16.078673 26114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:48:16.078822 26114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:48:16.078997 26114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:48:16.088310 26114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:34487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=39615
--tserver_master_addrs=127.9.166.62:39557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:48:16.089542 26114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:48:16.090880 26114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:48:16.103513 26122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:16.103559 26120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:16.103585 26119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:48:16.104725 26114 server_base.cc:1034] running on GCE node
I20260621 02:48:16.105299 26114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:48:16.106223 26114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:48:16.107481 26114 hybrid_clock.cc:648] HybridClock initialized: now 1782010096107433 us; error 38 us; skew 500 ppm
I20260621 02:48:16.107785 26114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:48:16 dist-test-slave-hvgn krb5kdc[25342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010096, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:48:16.110867 26114 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:48:16.112353 26114 webserver.cc:460] Webserver started at http://127.9.166.2:39615/ using document root <none> and password file <none>
I20260621 02:48:16.113065 26114 fs_manager.cc:359] Metadata directory not provided
I20260621 02:48:16.113219 26114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:48:16.118625 26114 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 02:48:16.121389 26129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:16.122637 26114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:48:16.122809 26114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "7eb4fd2059544a96b0498bfd25216fb9"
format_stamp: "Formatted at 2026-06-21 02:47:53 on dist-test-slave-hvgn"
server_key: "d8040508739196c6411f8ea82e840ee6"
server_key_iv: "bd1b623573f43f7f4146fa117fc2f2eb"
server_key_version: "encryptionkey@0"
I20260621 02:48:16.123831 26114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:48:16.154403 26114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:48:16.158854 26114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:48:16.159155 26114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:48:16.160148 26114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:48:16.161460 26114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:48:16.161587 26114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:16.161710 26114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:48:16.161794 26114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:16.196096 26114 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:34487
I20260621 02:48:16.196151 26241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:34487 every 8 connection(s)
I20260621 02:48:16.197471 26114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:48:16.201699 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 26114
I20260621 02:48:16.202543 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:40409
--local_ip_for_outbound_sockets=127.9.166.3
--tserver_master_addrs=127.9.166.62:39557
--webserver_port=39907
--webserver_interface=127.9.166.3
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 21 02:48:16 dist-test-slave-hvgn krb5kdc[25342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010096, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:48:16.221915 26112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:16.199908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:55647 (local address 127.9.166.62:39557)
0621 02:48:16.200083 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:16.200087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:16.201358 (+ 1271us) server_negotiation.cc:407] Connection header received
0621 02:48:16.204578 (+ 3220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:16.204583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:16.204664 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:16.204742 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:48:16.207048 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.207910 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.210674 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.211084 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.211181 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:16.215841 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:48:16.215846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:48:16.215850 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:48:16.215884 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:48:16.218510 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:16.219557 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:16.219561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:16.219563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:16.219679 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:16.220193 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:16.220196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:16.220198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:16.220451 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:48:16.220573 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:16.221505 (+ 932us) server_negotiation.cc:299] Negotiation successful
0621 02:48:16.221676 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260621 02:48:16.223717 26244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:16.200330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.2:55647)
0621 02:48:16.201177 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:16.201231 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0621 02:48:16.204225 (+ 2994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:48:16.205054 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:48:16.205070 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:48:16.205940 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:48:16.206789 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:16.206830 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.208091 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.208101 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:16.210518 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:16.210538 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.211342 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.211352 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:16.211638 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:16.212411 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:48:16.212436 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:48:16.215597 (+ 3161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:48:16.218651 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:48:16.218664 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:48:16.218684 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:48:16.219428 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:48:16.219794 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:48:16.219800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:48:16.219804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:48:16.220075 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:48:16.220597 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:48:16.220609 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:48:16.221072 (+ 463us) client_negotiation.cc:769] Sending connection context
0621 02:48:16.221426 (+ 354us) client_negotiation.cc:240] Negotiation successful
0621 02:48:16.222044 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":309,"threads_started":1}
I20260621 02:48:16.225414 26242 heartbeater.cc:347] Connected to a master server at 127.9.166.62:39557
I20260621 02:48:16.225826 26242 heartbeater.cc:464] Registering TS with master...
I20260621 02:48:16.226874 26242 heartbeater.cc:511] Master 127.9.166.62:39557 requested a full tablet report, sending...
I20260621 02:48:16.228741 25925 ts_manager.cc:194] Registered new tserver with Master: 7eb4fd2059544a96b0498bfd25216fb9 (127.9.166.2:34487)
I20260621 02:48:16.229568 25925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:55647
W20260621 02:48:16.545317 26245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:48:16.545576 26245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:48:16.545737 26245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:48:16.553630 26245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:48:16.553768 26245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:48:16.553960 26245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:48:16.563692 26245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:40409
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=39907
--tserver_master_addrs=127.9.166.62:39557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:48:16.564936 26245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:48:16.566251 26245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:48:16.579154 26250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:16.579210 26253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:16.579249 26251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:48:16.580178 26245 server_base.cc:1034] running on GCE node
I20260621 02:48:16.580744 26245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:48:16.581646 26245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:48:16.582841 26245 hybrid_clock.cc:648] HybridClock initialized: now 1782010096582776 us; error 61 us; skew 500 ppm
I20260621 02:48:16.583161 26245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:48:16 dist-test-slave-hvgn krb5kdc[25342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010096, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:48:16.586190 26245 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:48:16.587703 26245 webserver.cc:460] Webserver started at http://127.9.166.3:39907/ using document root <none> and password file <none>
I20260621 02:48:16.588505 26245 fs_manager.cc:359] Metadata directory not provided
I20260621 02:48:16.588660 26245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:48:16.594100 26245 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260621 02:48:16.596879 26260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:16.598111 26245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:48:16.598281 26245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "b14ec0114d2f4eaab780f5fc0425fbe9"
format_stamp: "Formatted at 2026-06-21 02:47:53 on dist-test-slave-hvgn"
server_key: "7fa7cc0f7fcd51d94c1ac874808cd8f3"
server_key_iv: "6da50083227690b0a8a2c65f84d0cf32"
server_key_version: "encryptionkey@0"
I20260621 02:48:16.599232 26245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:48:16.621104 26245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:48:16.624893 26245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:48:16.625155 26245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:48:16.626119 26245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:48:16.627485 26245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:48:16.627616 26245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:16.627734 26245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:48:16.627821 26245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:16.661321 26245 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:40409
I20260621 02:48:16.661405 26372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:40409 every 8 connection(s)
I20260621 02:48:16.662737 26245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:48:16.672209 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 26245
Jun 21 02:48:16 dist-test-slave-hvgn krb5kdc[25342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010096, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:48:16.686688 26112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:16.665291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:43537 (local address 127.9.166.62:39557)
0621 02:48:16.665493 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:16.665497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:16.666585 (+ 1088us) server_negotiation.cc:407] Connection header received
0621 02:48:16.668480 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:16.668486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:16.668574 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:16.668683 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:48:16.670189 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.670922 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.672645 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.673360 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.673521 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:16.679130 (+ 5609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:48:16.679178 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:48:16.679183 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:48:16.679221 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:48:16.683027 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:16.684106 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:16.684112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:16.684116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:16.684264 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:16.684893 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:16.684897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:16.684900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:16.685173 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:48:16.685286 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:16.686247 (+ 961us) server_negotiation.cc:299] Negotiation successful
0621 02:48:16.686421 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260621 02:48:16.688279 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:16.665688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.3:43537)
0621 02:48:16.666410 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:16.666463 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 02:48:16.668203 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:48:16.668946 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:48:16.668956 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:48:16.669491 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:48:16.670018 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:16.670036 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.671076 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.671086 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:16.672491 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:16.672503 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.673924 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.673934 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:16.674386 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:16.675443 (+ 1057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:48:16.675491 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:48:16.678830 (+ 3339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:48:16.683195 (+ 4365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:48:16.683202 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:48:16.683217 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:48:16.683944 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:48:16.684438 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:48:16.684445 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:48:16.684453 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:48:16.684718 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:48:16.685309 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:48:16.685320 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:48:16.685822 (+ 502us) client_negotiation.cc:769] Sending connection context
0621 02:48:16.686154 (+ 332us) client_negotiation.cc:240] Negotiation successful
0621 02:48:16.686748 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":294,"threads_started":1}
I20260621 02:48:16.689947 26373 heartbeater.cc:347] Connected to a master server at 127.9.166.62:39557
I20260621 02:48:16.690282 26373 heartbeater.cc:464] Registering TS with master...
I20260621 02:48:16.691049 26373 heartbeater.cc:511] Master 127.9.166.62:39557 requested a full tablet report, sending...
I20260621 02:48:16.692790 25925 ts_manager.cc:194] Registered new tserver with Master: b14ec0114d2f4eaab780f5fc0425fbe9 (127.9.166.3:40409)
I20260621 02:48:16.693305 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:16.673271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43690 (local address 127.9.166.62:39557)
0621 02:48:16.676407 (+ 3136us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:16.676427 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:16.676906 (+ 479us) server_negotiation.cc:407] Connection header received
0621 02:48:16.677146 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:16.677152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:16.677242 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:16.677356 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:48:16.679604 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.680941 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.683118 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.683762 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.683885 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:16.687120 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:48:16.687127 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:48:16.687131 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:48:16.687166 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:48:16.690460 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:16.691129 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:16.691136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:16.691141 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:16.691278 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:16.691735 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:16.691739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:16.691743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:16.692086 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:48:16.692351 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:16.692671 (+ 320us) server_negotiation.cc:299] Negotiation successful
0621 02:48:16.692893 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3013,"thread_start_us":2870,"threads_started":1}
I20260621 02:48:16.694000 25925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:43537
I20260621 02:48:16.695720 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:48:16.701983 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:16.696689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43694 (local address 127.9.166.62:39557)
0621 02:48:16.696852 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:16.696855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:16.697280 (+ 425us) server_negotiation.cc:407] Connection header received
0621 02:48:16.697457 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:16.697461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:16.697583 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:16.697694 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:48:16.698119 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.698945 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.700445 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.700842 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.700940 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:16.701353 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:48:16.701537 (+ 184us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0621 02:48:16.701796 (+ 259us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43694: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260621 02:48:16.702116 26377 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43694: token expired
W20260621 02:48:16.702692 25848 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.166.62:39557)
I20260621 02:48:16.715490 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:16.703162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43698 (local address 127.9.166.62:39557)
0621 02:48:16.703313 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:16.703318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:16.703364 (+ 46us) server_negotiation.cc:407] Connection header received
0621 02:48:16.703591 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:16.703601 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:16.703653 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:16.703747 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:48:16.705048 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.705701 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.707040 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.707486 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.707604 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:16.710178 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:48:16.710183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:48:16.710186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:48:16.710214 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:48:16.712822 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:16.713484 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:16.713491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:16.713495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:16.713640 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:16.713997 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:16.714003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:16.714006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:16.714310 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:48:16.714558 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:16.714881 (+ 323us) server_negotiation.cc:299] Negotiation successful
0621 02:48:16.715107 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 02:48:16.718869 25925 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:48:16.726373 25925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:48:16.749166 26387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:16.740315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:40409 (local address 127.0.0.1:51158)
0621 02:48:16.740829 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:16.740847 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:48:16.740963 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:48:16.742132 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:48:16.742139 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:48:16.742164 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:48:16.742605 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:16.742617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.744831 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.744836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:16.746671 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:16.746683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.748269 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.748275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:16.748475 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:16.748529 (+ 54us) client_negotiation.cc:769] Sending connection context
0621 02:48:16.748644 (+ 115us) client_negotiation.cc:240] Negotiation successful
0621 02:48:16.748816 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":1117,"threads_started":1}
I20260621 02:48:16.750592 26384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:16.737926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:34487 (local address 127.0.0.1:56010)
0621 02:48:16.738973 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:16.739023 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0621 02:48:16.739227 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:48:16.740880 (+ 1653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:48:16.740887 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:48:16.740923 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:48:16.741417 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:16.741435 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.743598 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.743603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:16.747944 (+ 4341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:16.747958 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.749802 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.749810 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:16.749967 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:16.749998 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 02:48:16.750063 (+ 65us) client_negotiation.cc:240] Negotiation successful
0621 02:48:16.750191 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":1938,"thread_start_us":2013,"threads_started":1}
I20260621 02:48:16.751087 26388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:16.740911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51158 (local address 127.9.166.3:40409)
0621 02:48:16.741624 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:16.741632 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:16.741662 (+ 30us) server_negotiation.cc:407] Connection header received
0621 02:48:16.741755 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:16.741760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:16.741912 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:16.742794 (+ 882us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:48:16.743109 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.744676 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.746838 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.748040 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.749928 (+ 1888us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:16.750179 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:16.750249 (+ 70us) server_negotiation.cc:299] Negotiation successful
0621 02:48:16.750410 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":618,"threads_started":1}
I20260621 02:48:16.751116 26383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:16.738147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:36415 (local address 127.0.0.1:55778)
0621 02:48:16.740219 (+ 2072us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:16.740238 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:48:16.740339 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:48:16.741092 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:48:16.741096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:48:16.741115 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:48:16.741479 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:16.741490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.744176 (+ 2686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.744181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:16.746423 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:16.746437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.749354 (+ 2917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.749360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:16.749548 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:16.749585 (+ 37us) client_negotiation.cc:769] Sending connection context
0621 02:48:16.749649 (+ 64us) client_negotiation.cc:240] Negotiation successful
0621 02:48:16.750823 (+ 1174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1939,"thread_start_us":420,"threads_started":1}
I20260621 02:48:16.751611 26385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:16.738245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56010 (local address 127.9.166.2:34487)
0621 02:48:16.740212 (+ 1967us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:16.740221 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:16.740254 (+ 33us) server_negotiation.cc:407] Connection header received
0621 02:48:16.740359 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:16.740367 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:16.740627 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:16.741637 (+ 1010us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:48:16.741879 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.743461 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.748124 (+ 4663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.749544 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.750819 (+ 1275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:16.751012 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:16.751079 (+ 67us) server_negotiation.cc:299] Negotiation successful
0621 02:48:16.751251 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1785,"thread_start_us":1677,"threads_started":1}
I20260621 02:48:16.752586 26386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:16.738737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.9.166.1:36415)
0621 02:48:16.740521 (+ 1784us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:16.740529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:16.740555 (+ 26us) server_negotiation.cc:407] Connection header received
0621 02:48:16.740642 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:16.740652 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:16.740879 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:16.741664 (+ 785us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:48:16.741879 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.744002 (+ 2123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.747629 (+ 3627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.749120 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.751068 (+ 1948us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:16.751269 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:16.751331 (+ 62us) server_negotiation.cc:299] Negotiation successful
0621 02:48:16.751510 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1627,"thread_start_us":2475,"threads_started":1}
I20260621 02:48:16.754989 26177 tablet_service.cc:1467] Processing CreateTablet for tablet f7097773baab41ea81649c2e525c7872 (DEFAULT_TABLE table=test-table [id=d8264a9d21324fac8df4915442037b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:48:16.755056 26308 tablet_service.cc:1467] Processing CreateTablet for tablet f7097773baab41ea81649c2e525c7872 (DEFAULT_TABLE table=test-table [id=d8264a9d21324fac8df4915442037b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:48:16.756315 26177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7097773baab41ea81649c2e525c7872. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:48:16.755827 26045 tablet_service.cc:1467] Processing CreateTablet for tablet f7097773baab41ea81649c2e525c7872 (DEFAULT_TABLE table=test-table [id=d8264a9d21324fac8df4915442037b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:48:16.756417 26308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7097773baab41ea81649c2e525c7872. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:48:16.757122 26045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7097773baab41ea81649c2e525c7872. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:48:16.768731 26389 tablet_bootstrap.cc:492] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9: Bootstrap starting.
I20260621 02:48:16.771253 26390 tablet_bootstrap.cc:492] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9: Bootstrap starting.
I20260621 02:48:16.771876 26389 tablet_bootstrap.cc:654] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9: Neither blocks nor log segments found. Creating new log.
I20260621 02:48:16.772977 26391 tablet_bootstrap.cc:492] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524: Bootstrap starting.
I20260621 02:48:16.773648 26389 log.cc:824] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9: Log is configured to *not* fsync() on all Append() calls
I20260621 02:48:16.775362 26390 tablet_bootstrap.cc:654] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9: Neither blocks nor log segments found. Creating new log.
I20260621 02:48:16.776042 26391 tablet_bootstrap.cc:654] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524: Neither blocks nor log segments found. Creating new log.
I20260621 02:48:16.776563 26390 log.cc:824] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9: Log is configured to *not* fsync() on all Append() calls
I20260621 02:48:16.777215 26391 log.cc:824] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524: Log is configured to *not* fsync() on all Append() calls
I20260621 02:48:16.777531 26389 tablet_bootstrap.cc:492] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9: No bootstrap required, opened a new log
I20260621 02:48:16.777909 26389 ts_tablet_manager.cc:1397] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260621 02:48:16.779531 26110 heartbeater.cc:503] Master 127.9.166.62:39557 was elected leader, sending a full tablet report...
I20260621 02:48:16.779850 26390 tablet_bootstrap.cc:492] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9: No bootstrap required, opened a new log
I20260621 02:48:16.780102 26390 ts_tablet_manager.cc:1397] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260621 02:48:16.780297 26391 tablet_bootstrap.cc:492] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524: No bootstrap required, opened a new log
I20260621 02:48:16.780568 26391 ts_tablet_manager.cc:1397] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 02:48:16.786506 26389 raft_consensus.cc:348] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 } } peers { permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 } } peers { permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 } }
I20260621 02:48:16.786921 26389 raft_consensus.cc:374] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:48:16.787040 26389 raft_consensus.cc:729] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4fd2059544a96b0498bfd25216fb9, State: Initialized, Role: FOLLOWER
I20260621 02:48:16.787866 26389 consensus_queue.cc:260] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 } } peers { permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 } } peers { permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 } }
I20260621 02:48:16.789054 26242 heartbeater.cc:503] Master 127.9.166.62:39557 was elected leader, sending a full tablet report...
I20260621 02:48:16.789685 26389 ts_tablet_manager.cc:1428] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260621 02:48:16.790597 26390 raft_consensus.cc:348] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 } } peers { permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 } } peers { permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 } }
I20260621 02:48:16.790940 26390 raft_consensus.cc:374] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:48:16.791046 26390 raft_consensus.cc:729] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b14ec0114d2f4eaab780f5fc0425fbe9, State: Initialized, Role: FOLLOWER
I20260621 02:48:16.791683 26390 consensus_queue.cc:260] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 } } peers { permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 } } peers { permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 } }
I20260621 02:48:16.791882 26391 raft_consensus.cc:348] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 } } peers { permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 } } peers { permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 } }
I20260621 02:48:16.792318 26391 raft_consensus.cc:374] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:48:16.792450 26391 raft_consensus.cc:729] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8e3c5abc7e4551ae9848742981a524, State: Initialized, Role: FOLLOWER
I20260621 02:48:16.793035 26373 heartbeater.cc:503] Master 127.9.166.62:39557 was elected leader, sending a full tablet report...
I20260621 02:48:16.793037 26391 consensus_queue.cc:260] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 } } peers { permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 } } peers { permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 } }
I20260621 02:48:16.794499 26390 ts_tablet_manager.cc:1428] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260621 02:48:16.795861 26391 ts_tablet_manager.cc:1428] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260621 02:48:16.914358 26374 tablet.cc:2369] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:48:16.949517 26243 tablet.cc:2369] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:48:16.964571 26396 raft_consensus.cc:482] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:48:16.964836 26396 raft_consensus.cc:504] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 } } peers { permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 } } peers { permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 } }
I20260621 02:48:16.966441 26396 leader_election.cc:288] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4fd2059544a96b0498bfd25216fb9 (127.9.166.2:34487), cc8e3c5abc7e4551ae9848742981a524 (127.9.166.1:36415)
I20260621 02:48:16.973836 26385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:16.966904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:33285 (local address 127.9.166.2:34487)
0621 02:48:16.967116 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:16.967123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:16.967235 (+ 112us) server_negotiation.cc:407] Connection header received
0621 02:48:16.967496 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:16.967505 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:16.967609 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:16.967721 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:48:16.968209 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.969191 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.971557 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.972657 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.972792 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:16.972913 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:16.973385 (+ 472us) server_negotiation.cc:299] Negotiation successful
0621 02:48:16.973522 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:48:16.973964 26386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:16.966910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:54485 (local address 127.9.166.1:36415)
0621 02:48:16.967116 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:16.967123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:16.968016 (+ 893us) server_negotiation.cc:407] Connection header received
0621 02:48:16.968188 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:16.968193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:16.968269 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:16.968366 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:48:16.968828 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.969681 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.971917 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.973070 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.973211 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:16.973303 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:16.973593 (+ 290us) server_negotiation.cc:299] Negotiation successful
0621 02:48:16.973712 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:48:16.974049 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:16.966970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:36415 (local address 127.9.166.3:54485)
0621 02:48:16.967912 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:16.967929 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:48:16.968048 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:48:16.968387 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:48:16.968390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:48:16.968409 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:48:16.968691 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:16.968698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.969830 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.969836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:16.971752 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:16.971773 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.973267 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.973273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:16.973432 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:16.973463 (+ 31us) client_negotiation.cc:769] Sending connection context
0621 02:48:16.973549 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 02:48:16.973668 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":182,"thread_start_us":386,"threads_started":1}
I20260621 02:48:16.974398 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:16.966949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:34487 (local address 127.9.166.3:33285)
0621 02:48:16.967124 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:16.967143 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:48:16.967269 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:48:16.967772 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:48:16.967820 (+ 48us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:48:16.967850 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:48:16.968073 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:16.968086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.969351 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.969355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:16.971378 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:16.971418 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:16.972883 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:16.972889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:16.973052 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:16.973084 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 02:48:16.973187 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 02:48:16.974125 (+ 938us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260621 02:48:16.975068 26065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7097773baab41ea81649c2e525c7872" candidate_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8e3c5abc7e4551ae9848742981a524" is_pre_election: true
I20260621 02:48:16.975374 26197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7097773baab41ea81649c2e525c7872" candidate_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4fd2059544a96b0498bfd25216fb9" is_pre_election: true
I20260621 02:48:16.975625 26065 raft_consensus.cc:2454] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b14ec0114d2f4eaab780f5fc0425fbe9 in term 0.
I20260621 02:48:16.975883 26197 raft_consensus.cc:2454] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b14ec0114d2f4eaab780f5fc0425fbe9 in term 0.
I20260621 02:48:16.976342 26261 leader_election.cc:302] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b14ec0114d2f4eaab780f5fc0425fbe9, cc8e3c5abc7e4551ae9848742981a524; no voters:
I20260621 02:48:16.976735 26396 raft_consensus.cc:2781] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:48:16.976917 26396 raft_consensus.cc:482] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:48:16.977036 26396 raft_consensus.cc:3037] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:48:16.979163 26396 raft_consensus.cc:504] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 } } peers { permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 } } peers { permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 } }
I20260621 02:48:16.979883 26396 leader_election.cc:288] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [CANDIDATE]: Term 1 election: Requested vote from peers 7eb4fd2059544a96b0498bfd25216fb9 (127.9.166.2:34487), cc8e3c5abc7e4551ae9848742981a524 (127.9.166.1:36415)
I20260621 02:48:16.980170 26197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7097773baab41ea81649c2e525c7872" candidate_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4fd2059544a96b0498bfd25216fb9"
I20260621 02:48:16.980388 26197 raft_consensus.cc:3037] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:48:16.980388 26065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7097773baab41ea81649c2e525c7872" candidate_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8e3c5abc7e4551ae9848742981a524"
I20260621 02:48:16.980576 26065 raft_consensus.cc:3037] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:48:16.982573 26197 raft_consensus.cc:2454] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b14ec0114d2f4eaab780f5fc0425fbe9 in term 1.
I20260621 02:48:16.982725 26065 raft_consensus.cc:2454] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b14ec0114d2f4eaab780f5fc0425fbe9 in term 1.
I20260621 02:48:16.982990 26264 leader_election.cc:302] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4fd2059544a96b0498bfd25216fb9, b14ec0114d2f4eaab780f5fc0425fbe9; no voters:
I20260621 02:48:16.983306 26396 raft_consensus.cc:2781] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:48:16.983764 26396 raft_consensus.cc:686] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [term 1 LEADER]: Becoming Leader. State: Replica: b14ec0114d2f4eaab780f5fc0425fbe9, State: Running, Role: LEADER
I20260621 02:48:16.984227 26396 consensus_queue.cc:237] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 } } peers { permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 } } peers { permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 } }
I20260621 02:48:16.988617 25925 catalog_manager.cc:5495] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b14ec0114d2f4eaab780f5fc0425fbe9 (127.9.166.3). New cstate: current_term: 1 leader_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b14ec0114d2f4eaab780f5fc0425fbe9" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 40409 } health_report { overall_health: HEALTHY } } }
W20260621 02:48:16.997816 26111 tablet.cc:2369] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:48:17.044607 26388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:17.039223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51162 (local address 127.9.166.3:40409)
0621 02:48:17.039443 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:17.039449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:17.039468 (+ 19us) server_negotiation.cc:407] Connection header received
0621 02:48:17.039684 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:17.039688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:17.039774 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:17.039937 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:48:17.040291 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:17.041010 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:17.042396 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:17.042805 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:17.042957 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:17.043495 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:48:17.044001 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:48:17.044161 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:17.044258 (+ 97us) server_negotiation.cc:299] Negotiation successful
0621 02:48:17.044341 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260621 02:48:17.052431 26065 raft_consensus.cc:1261] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524 [term 1 FOLLOWER]: Refusing update from remote peer b14ec0114d2f4eaab780f5fc0425fbe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:48:17.052475 26197 raft_consensus.cc:1261] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9 [term 1 FOLLOWER]: Refusing update from remote peer b14ec0114d2f4eaab780f5fc0425fbe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:48:17.053480 26399 consensus_queue.cc:1035] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4fd2059544a96b0498bfd25216fb9" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 34487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:48:17.053838 26396 consensus_queue.cc:1035] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8e3c5abc7e4551ae9848742981a524" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:48:17.066900 26402 mvcc.cc:205] Tried to move back new op lower bound from 7299113357517266944 to 7299113357250871296. Current Snapshot: MvccSnapshot[applied={T|T < 7299113357517266944}]
I20260621 02:48:17.068573 26403 mvcc.cc:205] Tried to move back new op lower bound from 7299113357517266944 to 7299113357250871296. Current Snapshot: MvccSnapshot[applied={T|T < 7299113357517266944}]
I20260621 02:48:17.079331 25925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:48:17.079629 25925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:48:17.082948 25925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c: Sending DeleteTablet for 3 replicas of tablet f7097773baab41ea81649c2e525c7872
I20260621 02:48:17.083994 26045 tablet_service.cc:1514] Processing DeleteTablet for tablet f7097773baab41ea81649c2e525c7872 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:48:17 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:55778
I20260621 02:48:17.084007 26308 tablet_service.cc:1514] Processing DeleteTablet for tablet f7097773baab41ea81649c2e525c7872 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:48:17 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:51158
I20260621 02:48:17.084088 26177 tablet_service.cc:1514] Processing DeleteTablet for tablet f7097773baab41ea81649c2e525c7872 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:48:17 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:56010
I20260621 02:48:17.085471 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 25962
I20260621 02:48:17.096704 26412 tablet_replica.cc:330] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9: stopping tablet replica
I20260621 02:48:17.097256 26411 tablet_replica.cc:330] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524: stopping tablet replica
I20260621 02:48:17.097689 26413 tablet_replica.cc:330] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9: stopping tablet replica
I20260621 02:48:17.098119 26412 raft_consensus.cc:2229] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:48:17.098112 26413 raft_consensus.cc:2229] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:48:17.098378 26411 raft_consensus.cc:2229] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:48:17.098663 26413 raft_consensus.cc:2258] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:48:17.099316 26412 raft_consensus.cc:2258] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:48:17.099634 26411 raft_consensus.cc:2258] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:48:17.101284 26413 ts_tablet_manager.cc:1905] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:48:17.102149 26411 ts_tablet_manager.cc:1905] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:48:17.108489 26413 ts_tablet_manager.cc:1918] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:48:17.108587 26386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:17.087153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55780 (local address 127.9.166.1:36415)
0621 02:48:17.087359 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:17.087367 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:17.087414 (+ 47us) server_negotiation.cc:407] Connection header received
0621 02:48:17.087479 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:17.087486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:17.087616 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:17.087749 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:48:17.090316 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:17.091787 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:17.094232 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:17.095107 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:17.095251 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:17.100088 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:48:17.100102 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:48:17.100119 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:48:17.100159 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:48:17.104353 (+ 4194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:17.105272 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:17.105281 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:17.105284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:17.105461 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:17.105928 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:17.105933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:17.105936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:17.106429 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:48:17.106748 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:17.107920 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0621 02:48:17.108147 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:48:17.109242 26105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:48:17.109910 26412 ts_tablet_manager.cc:1905] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:48:17.110332 26413 log.cc:1196] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/f7097773baab41ea81649c2e525c7872
I20260621 02:48:17.111155 26413 ts_tablet_manager.cc:1939] T f7097773baab41ea81649c2e525c7872 P 7eb4fd2059544a96b0498bfd25216fb9: Deleting consensus metadata
I20260621 02:48:17.113514 25912 catalog_manager.cc:4841] TS 7eb4fd2059544a96b0498bfd25216fb9 (127.9.166.2:34487): tablet f7097773baab41ea81649c2e525c7872 (table test-table [id=d8264a9d21324fac8df4915442037b3d]) successfully deleted
I20260621 02:48:17.117492 26412 ts_tablet_manager.cc:1918] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:48:17.117650 26412 log.cc:1196] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/f7097773baab41ea81649c2e525c7872
I20260621 02:48:17.118172 26412 ts_tablet_manager.cc:1939] T f7097773baab41ea81649c2e525c7872 P b14ec0114d2f4eaab780f5fc0425fbe9: Deleting consensus metadata
I20260621 02:48:17.119804 25912 catalog_manager.cc:4841] TS b14ec0114d2f4eaab780f5fc0425fbe9 (127.9.166.3:40409): tablet f7097773baab41ea81649c2e525c7872 (table test-table [id=d8264a9d21324fac8df4915442037b3d]) successfully deleted
I20260621 02:48:17.211325 26411 ts_tablet_manager.cc:1918] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:48:17.211624 26411 log.cc:1196] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/f7097773baab41ea81649c2e525c7872
I20260621 02:48:17.212057 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 25962
I20260621 02:48:17.212212 26411 ts_tablet_manager.cc:1939] T f7097773baab41ea81649c2e525c7872 P cc8e3c5abc7e4551ae9848742981a524: Deleting consensus metadata
I20260621 02:48:17.236359 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 26114
W20260621 02:48:17.236400 25909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.166.1:36415 (error 108)
I20260621 02:48:17.238897 26383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:17.238342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:36415 (local address 127.0.0.1:55788)
0621 02:48:17.238592 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:17.238723 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:36415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260621 02:48:17.239152 25909 catalog_manager.cc:4584] TS cc8e3c5abc7e4551ae9848742981a524 (127.9.166.1:36415): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7097773baab41ea81649c2e525c7872: Network error: Client connection negotiation failed: client connection to 127.9.166.1:36415: connect: Connection refused (error 111)
I20260621 02:48:17.252800 26385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:17.237353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56014 (local address 127.9.166.2:34487)
0621 02:48:17.237999 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:17.238006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:17.238024 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:48:17.238093 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:17.238098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:17.238205 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:17.238313 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:48:17.239885 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:17.241029 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:17.242398 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:17.243001 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:17.243117 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:17.245777 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:48:17.245794 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:48:17.245815 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:48:17.245860 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:48:17.249574 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:17.250422 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:17.250431 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:17.250435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:17.250576 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:17.250956 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:17.250961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:17.250971 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:17.251332 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:48:17.251701 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:17.252211 (+ 510us) server_negotiation.cc:299] Negotiation successful
0621 02:48:17.252423 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513}
I20260621 02:48:17.253655 26237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:48:17.292835 26383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:17.292292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:36415 (local address 127.0.0.1:55794)
0621 02:48:17.292556 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:17.292664 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:36415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260621 02:48:17.294044 26383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:17.293631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:36415 (local address 127.0.0.1:55802)
0621 02:48:17.293797 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:17.293899 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:36415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260621 02:48:17.350322 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 26114
I20260621 02:48:17.354974 26383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:17.354368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:36415 (local address 127.0.0.1:55804)
0621 02:48:17.354630 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:17.354745 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:36415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260621 02:48:17.356138 26383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:17.355735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:36415 (local address 127.0.0.1:55806)
0621 02:48:17.355907 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:17.355994 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:36415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260621 02:48:17.367761 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 26245
I20260621 02:48:17.382233 26388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:17.368536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51170 (local address 127.9.166.3:40409)
0621 02:48:17.368718 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:17.368731 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:17.368814 (+ 83us) server_negotiation.cc:407] Connection header received
0621 02:48:17.369026 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:17.369040 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:17.369112 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:17.369224 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:48:17.370576 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:17.371377 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:17.372746 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:17.373207 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:17.373346 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:17.376098 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:48:17.376110 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:48:17.376126 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:48:17.376160 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:48:17.379162 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:17.379937 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:17.379948 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:17.379955 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:17.380119 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:17.380500 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:17.380510 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:17.380512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:17.380836 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:48:17.381171 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:17.381469 (+ 298us) server_negotiation.cc:299] Negotiation successful
0621 02:48:17.381682 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:48:17.383191 26368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:48:17.436982 26383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:17.436415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:36415 (local address 127.0.0.1:55820)
0621 02:48:17.436654 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:17.436777 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:36415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260621 02:48:17.438164 26383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:17.437732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:36415 (local address 127.0.0.1:55834)
0621 02:48:17.437941 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:17.438035 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:36415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260621 02:48:17.506527 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 26245
I20260621 02:48:17.526189 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 25892
I20260621 02:48:17.527462 25955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:48:17.642314 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 25892
I20260621 02:48:17.642386 26383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:17.641920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:36415 (local address 127.0.0.1:55838)
0621 02:48:17.642126 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:17.642215 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:36415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260621 02:48:37.664140 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:39557
--webserver_interface=127.9.166.62
--webserver_port=43155
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:39557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:48:37.993690 26419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:48:37.994058 26419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:48:37.994220 26419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:48:38.001600 26419 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260621 02:48:38.001740 26419 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260621 02:48:38.001830 26419 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260621 02:48:38.001931 26419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:48:38.002038 26419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:48:38.002159 26419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:48:38.002247 26419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:48:38.002326 26419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:48:38.012522 26419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:39557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:39557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=43155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:48:38.013955 26419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:48:38.015288 26419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:48:38.027166 26424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:38.027132 26425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:38.027222 26427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:48:38.028278 26419 server_base.cc:1034] running on GCE node
I20260621 02:48:38.029183 26419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:48:38.030669 26419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:48:38.031935 26419 hybrid_clock.cc:648] HybridClock initialized: now 1782010118031858 us; error 54 us; skew 500 ppm
I20260621 02:48:38.032308 26419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:48:38 dist-test-slave-hvgn krb5kdc[25342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010118, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:48:38.041262 26419 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:48:38.042912 26419 webserver.cc:460] Webserver started at http://127.9.166.62:43155/ using document root <none> and password file <none>
I20260621 02:48:38.043740 26419 fs_manager.cc:359] Metadata directory not provided
I20260621 02:48:38.043876 26419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:48:38.049346 26419 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 02:48:38.052402 26434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:38.053879 26419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:48:38.054078 26419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c"
format_stamp: "Formatted at 2026-06-21 02:47:52 on dist-test-slave-hvgn"
server_key: "07184a62abd3b3472b81ba7c8159d7da"
server_key_iv: "0a85f39fd3133200e950f79d31052683"
server_key_version: "encryptionkey@0"
I20260621 02:48:38.055095 26419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:48:38.067256 26419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:48:38.071255 26419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:48:38.071563 26419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:48:38.090039 26419 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:39557
I20260621 02:48:38.090077 26485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:39557 every 8 connection(s)
I20260621 02:48:38.091629 26419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:48:38.096895 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 26419
I20260621 02:48:38.097761 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:36415
--local_ip_for_outbound_sockets=127.9.166.1
--tserver_master_addrs=127.9.166.62:39557
--webserver_port=34801
--webserver_interface=127.9.166.1
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:48:38.100008 26486 sys_catalog.cc:263] Verifying existing consensus state
I20260621 02:48:38.103225 26486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c: Bootstrap starting.
I20260621 02:48:38.133365 26486 log.cc:824] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:48:38.153848 26486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260621 02:48:38.154546 26486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c: Bootstrap complete.
I20260621 02:48:38.169726 26486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39557 } }
I20260621 02:48:38.171010 26486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9bc9b5c7d043c9a5b2d36105b1414c, State: Initialized, Role: FOLLOWER
I20260621 02:48:38.171749 26486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39557 } }
I20260621 02:48:38.172024 26486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:48:38.172147 26486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:48:38.172325 26486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 2 FOLLOWER]: Advancing to term 3
I20260621 02:48:38.177738 26486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39557 } }
I20260621 02:48:38.178311 26486 leader_election.cc:302] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a9bc9b5c7d043c9a5b2d36105b1414c; no voters:
I20260621 02:48:38.178972 26486 leader_election.cc:288] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260621 02:48:38.179117 26490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 3 FOLLOWER]: Leader election won for term 3
I20260621 02:48:38.180451 26490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [term 3 LEADER]: Becoming Leader. State: Replica: 6a9bc9b5c7d043c9a5b2d36105b1414c, State: Running, Role: LEADER
I20260621 02:48:38.180979 26486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:48:38.180985 26490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39557 } }
I20260621 02:48:38.183884 26491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39557 } } }
I20260621 02:48:38.184576 26491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [sys.catalog]: This master's current role is: LEADER
I20260621 02:48:38.183681 26492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a9bc9b5c7d043c9a5b2d36105b1414c. Latest consensus state: current_term: 3 leader_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9bc9b5c7d043c9a5b2d36105b1414c" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 39557 } } }
I20260621 02:48:38.185756 26492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c [sys.catalog]: This master's current role is: LEADER
I20260621 02:48:38.187023 26499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:48:38.198671 26499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:48:38.200124 26499 catalog_manager.cc:1259] Loaded cluster ID: 53b6fe39a5fe42dfa6faf338252f6d54
I20260621 02:48:38.200227 26499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:48:38.204365 26499 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:48:38.205863 26499 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c: Loaded TSK: 0
I20260621 02:48:38.206025 26499 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6a9bc9b5c7d043c9a5b2d36105b1414c: Loaded TSK: 1
F20260621 02:48:38.207108 26499 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb9945fd9ad google::LogMessage::Fail() at ??:0
@ 0x7fb99460088a google::LogMessage::SendToLog() at ??:0
@ 0x7fb9945fd4de google::LogMessage::Flush() at ??:0
@ 0x7fb9945fea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260621 02:48:38.216858 26507 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fb9983617d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb99835073e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb9aa02c1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb9aa02b783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb9aa0b72a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fb9aa0b70c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fb9aa1015e9 std::function<>::operator()() at ??:0
@ 0x7fb9aa02fcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb9aa02dc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb9aa0b5610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb9aa0b5439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb997249345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb99724c91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb99724c799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb997227eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb99a5016db start_thread at ??:0
@ 0x7fb99349761f clone at ??:0
W20260621 02:48:38.503582 26488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:48:38.503856 26488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:48:38.504019 26488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:48:38.511088 26488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:48:38.511204 26488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:48:38.511420 26488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:48:38.520998 26488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:36415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=34801
--tserver_master_addrs=127.9.166.62:39557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:48:38.522274 26488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:48:38.523761 26488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:48:38.537861 26534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:38.537818 26533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:38.537917 26536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:48:38.544082 26488 server_base.cc:1034] running on GCE node
I20260621 02:48:38.544731 26488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:48:38.545696 26488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:48:38.546926 26488 hybrid_clock.cc:648] HybridClock initialized: now 1782010118546875 us; error 47 us; skew 500 ppm
I20260621 02:48:38.547302 26488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:48:38 dist-test-slave-hvgn krb5kdc[25342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010118, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:48:38.550333 26488 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:48:38.551898 26488 webserver.cc:460] Webserver started at http://127.9.166.1:34801/ using document root <none> and password file <none>
I20260621 02:48:38.552729 26488 fs_manager.cc:359] Metadata directory not provided
I20260621 02:48:38.552883 26488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:48:38.558257 26488 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260621 02:48:38.561000 26543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:38.562709 26488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:48:38.562852 26488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "cc8e3c5abc7e4551ae9848742981a524"
format_stamp: "Formatted at 2026-06-21 02:47:52 on dist-test-slave-hvgn"
server_key: "908a1a3f5bb68722696ccc7526db3700"
server_key_iv: "1d2e95038339e8fd93e7240600292709"
server_key_version: "encryptionkey@0"
I20260621 02:48:38.564241 26488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:48:38.584214 26488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:48:38.588512 26488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:48:38.588788 26488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:48:38.589751 26488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:48:38.591032 26488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:48:38.591151 26488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:38.591297 26488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:48:38.591421 26488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:38.623189 26488 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:36415
I20260621 02:48:38.623276 26655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:36415 every 8 connection(s)
I20260621 02:48:38.624711 26488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:48:38.625789 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 26488
I20260621 02:48:38.626554 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:34487
--local_ip_for_outbound_sockets=127.9.166.2
--tserver_master_addrs=127.9.166.62:39557
--webserver_port=39615
--webserver_interface=127.9.166.2
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:48:38.629835 26658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:38.627805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.1:56867)
0621 02:48:38.628650 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:38.628915 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":353,"threads_started":1}
W20260621 02:48:38.630546 26656 heartbeater.cc:644] Failed to heartbeat to 127.9.166.62:39557 (0 consecutive failures): Network error: Failed to ping master at 127.9.166.62:39557: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
I20260621 02:48:38.631428 26658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:38.631084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.1:36573)
0621 02:48:38.631219 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:38.631285 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260621 02:48:38.632321 26658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:38.631991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.1:35537)
0621 02:48:38.632136 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:38.632217 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260621 02:48:38.632570 26656 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 02:48:38.966688 26659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:48:38.967022 26659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:48:38.967187 26659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:48:38.974635 26659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:48:38.974784 26659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:48:38.974967 26659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:48:38.984531 26659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:34487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=39615
--tserver_master_addrs=127.9.166.62:39557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:48:38.985806 26659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:48:38.987141 26659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:48:38.999670 26664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:38.999718 26665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:38.999629 26667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:48:39.000828 26659 server_base.cc:1034] running on GCE node
I20260621 02:48:39.001458 26659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:48:39.002351 26659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:48:39.003583 26659 hybrid_clock.cc:648] HybridClock initialized: now 1782010119003538 us; error 54 us; skew 500 ppm
I20260621 02:48:39.003895 26659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:48:39 dist-test-slave-hvgn krb5kdc[25342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010119, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:48:39.006970 26659 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:48:39.008466 26659 webserver.cc:460] Webserver started at http://127.9.166.2:39615/ using document root <none> and password file <none>
I20260621 02:48:39.009140 26659 fs_manager.cc:359] Metadata directory not provided
I20260621 02:48:39.009285 26659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:48:39.014596 26659 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260621 02:48:39.017307 26674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:39.018533 26659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:48:39.018697 26659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "7eb4fd2059544a96b0498bfd25216fb9"
format_stamp: "Formatted at 2026-06-21 02:47:53 on dist-test-slave-hvgn"
server_key: "d8040508739196c6411f8ea82e840ee6"
server_key_iv: "bd1b623573f43f7f4146fa117fc2f2eb"
server_key_version: "encryptionkey@0"
I20260621 02:48:39.019698 26659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:48:39.031764 26659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:48:39.036275 26659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:48:39.036567 26659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:48:39.037572 26659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:48:39.038935 26659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:48:39.039057 26659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:39.039191 26659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:48:39.039279 26659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:39.072736 26659 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:34487
I20260621 02:48:39.072824 26786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:34487 every 8 connection(s)
I20260621 02:48:39.074153 26659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:48:39.078830 26789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:39.077096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.2:57179)
0621 02:48:39.077793 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:39.077988 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":280,"threads_started":1}
W20260621 02:48:39.079607 26787 heartbeater.cc:644] Failed to heartbeat to 127.9.166.62:39557 (0 consecutive failures): Network error: Failed to ping master at 127.9.166.62:39557: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
I20260621 02:48:39.079950 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 26659
I20260621 02:48:39.080515 26789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:39.080084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.2:46615)
0621 02:48:39.080243 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:39.080374 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260621 02:48:39.080765 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:40409
--local_ip_for_outbound_sockets=127.9.166.3
--tserver_master_addrs=127.9.166.62:39557
--webserver_port=39907
--webserver_interface=127.9.166.3
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:43213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:48:39.081346 26789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:39.081050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.2:53335)
0621 02:48:39.081173 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:39.081243 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260621 02:48:39.081638 26787 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260621 02:48:39.405084 26790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:48:39.405355 26790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:48:39.405501 26790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:48:39.412673 26790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:48:39.412796 26790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:48:39.412981 26790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:48:39.422178 26790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:43213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:40409
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=39907
--tserver_master_addrs=127.9.166.62:39557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:48:39.423418 26790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:48:39.424763 26790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:48:39.437718 26796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:39.437753 26798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:39.437789 26795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:48:39.438591 26790 server_base.cc:1034] running on GCE node
I20260621 02:48:39.439121 26790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:48:39.440081 26790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:48:39.441311 26790 hybrid_clock.cc:648] HybridClock initialized: now 1782010119441268 us; error 37 us; skew 500 ppm
I20260621 02:48:39.441653 26790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:48:39 dist-test-slave-hvgn krb5kdc[25342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010119, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:48:39.444751 26790 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:48:39.446192 26790 webserver.cc:460] Webserver started at http://127.9.166.3:39907/ using document root <none> and password file <none>
I20260621 02:48:39.446808 26790 fs_manager.cc:359] Metadata directory not provided
I20260621 02:48:39.446908 26790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:48:39.452049 26790 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:48:39.456267 26805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:39.457686 26790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:48:39.457819 26790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "b14ec0114d2f4eaab780f5fc0425fbe9"
format_stamp: "Formatted at 2026-06-21 02:47:53 on dist-test-slave-hvgn"
server_key: "7fa7cc0f7fcd51d94c1ac874808cd8f3"
server_key_iv: "6da50083227690b0a8a2c65f84d0cf32"
server_key_version: "encryptionkey@0"
I20260621 02:48:39.458763 26790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:48:39.484336 26790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:48:39.488112 26790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:48:39.488345 26790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:48:39.489230 26790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:48:39.490443 26790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:48:39.490527 26790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:39.490608 26790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:48:39.490664 26790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:39.524094 26790 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:40409
I20260621 02:48:39.524161 26917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:40409 every 8 connection(s)
I20260621 02:48:39.525534 26790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:48:39.531052 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:39.528920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.3:48973)
0621 02:48:39.529838 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:39.530138 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":364,"threads_started":1}
W20260621 02:48:39.531692 26918 heartbeater.cc:644] Failed to heartbeat to 127.9.166.62:39557 (0 consecutive failures): Network error: Failed to ping master at 127.9.166.62:39557: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
I20260621 02:48:39.532449 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:39.532166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.3:34649)
0621 02:48:39.532291 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:39.532354 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260621 02:48:39.533097 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 26790
I20260621 02:48:39.533300 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:39.532918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.3:49053)
0621 02:48:39.533067 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:39.533158 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260621 02:48:39.533483 26918 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260621 02:48:39.633846 26922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:39.633050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.1:43645)
0621 02:48:39.633491 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:39.633595 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":246,"threads_started":1}
I20260621 02:48:40.082942 26923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:40.082183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.2:41099)
0621 02:48:40.082626 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:40.082719 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":223,"threads_started":1}
I20260621 02:48:40.535032 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:40.534155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.3:43443)
0621 02:48:40.534697 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:40.534800 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":267,"threads_started":1}
I20260621 02:48:40.635474 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:40.634729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.1:55439)
0621 02:48:40.635168 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:40.635246 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":211,"threads_started":1}
I20260621 02:48:41.084503 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:41.083814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.2:38117)
0621 02:48:41.084250 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:41.084328 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":214,"threads_started":1}
I20260621 02:48:41.536831 26927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:41.535902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.3:33597)
0621 02:48:41.536502 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:41.536616 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":350,"threads_started":1}
I20260621 02:48:41.636921 26928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:41.636258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.1:37649)
0621 02:48:41.636672 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:41.636747 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":215,"threads_started":1}
I20260621 02:48:42.086087 26929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:42.085388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.2:53023)
0621 02:48:42.085802 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:42.085879 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":223,"threads_started":1}
I20260621 02:48:42.538605 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:42.537694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.3:54073)
0621 02:48:42.538217 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:42.538316 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":304,"threads_started":1}
I20260621 02:48:42.638674 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:42.637862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.1:52763)
0621 02:48:42.638380 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:42.638479 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":306,"threads_started":1}
I20260621 02:48:43.087674 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:43.086920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.2:40009)
0621 02:48:43.087345 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:43.087467 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":219,"threads_started":1}
I20260621 02:48:43.540292 26933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:43.539615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.3:33883)
0621 02:48:43.540040 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:43.540120 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":207,"threads_started":1}
I20260621 02:48:43.640429 26934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:43.639650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.1:35963)
0621 02:48:43.640159 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:43.640239 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":249,"threads_started":1}
I20260621 02:48:44.089409 26935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:44.088701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.2:42081)
0621 02:48:44.089163 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:44.089240 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":260,"threads_started":1}
I20260621 02:48:44.541874 26936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:44.541182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.3:39821)
0621 02:48:44.541625 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:44.541702 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":224,"threads_started":1}
I20260621 02:48:44.642123 26937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:44.641302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.1:40915)
0621 02:48:44.641747 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:44.641902 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":233,"threads_started":1}
I20260621 02:48:45.091132 26938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:45.090348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.2:47987)
0621 02:48:45.090862 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:45.090951 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":313,"threads_started":1}
I20260621 02:48:45.543532 26939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:45.542710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.3:54223)
0621 02:48:45.543213 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:45.543319 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":266,"threads_started":1}
I20260621 02:48:45.644090 26940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:45.643132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.1:36601)
0621 02:48:45.643675 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:45.643792 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":277,"threads_started":1}
I20260621 02:48:46.092862 26941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:46.092103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.2:52793)
0621 02:48:46.092567 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:46.092648 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":219,"threads_started":1}
I20260621 02:48:46.545194 26942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:46.544412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.3:51409)
0621 02:48:46.544885 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:46.544988 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":225,"threads_started":1}
I20260621 02:48:46.645772 26943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:46.645006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.1:50223)
0621 02:48:46.645524 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:46.645607 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":293,"threads_started":1}
I20260621 02:48:47.094416 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:47.093747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.2:52409)
0621 02:48:47.094186 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:47.094262 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":228,"threads_started":1}
I20260621 02:48:47.547010 26945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:47.546168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.3:40927)
0621 02:48:47.546662 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:47.546768 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":233,"threads_started":1}
I20260621 02:48:47.647358 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:47.646651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.1:41559)
0621 02:48:47.647097 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:47.647174 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":223,"threads_started":1}
I20260621 02:48:48.096148 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:48.095334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.2:42131)
0621 02:48:48.095862 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:48.095964 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":295,"threads_started":1}
I20260621 02:48:48.548960 26948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:48.548010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.3:40347)
0621 02:48:48.548560 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:48.548672 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":263,"threads_started":1}
I20260621 02:48:48.649053 26949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:48.648378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.1:52237)
0621 02:48:48.648811 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:48.648892 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":243,"threads_started":1}
I20260621 02:48:49.097870 26950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:49.097201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.2:56833)
0621 02:48:49.097618 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:49.097695 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":227,"threads_started":1}
I20260621 02:48:49.550581 26951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:49.549870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.3:56841)
0621 02:48:49.550339 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:49.550422 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":234,"threads_started":1}
I20260621 02:48:49.650682 26952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:49.649969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.1:46295)
0621 02:48:49.650417 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:49.650506 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":197,"threads_started":1}
I20260621 02:48:50.099581 26953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:50.098808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.2:35173)
0621 02:48:50.099242 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:50.099320 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":219,"threads_started":1}
I20260621 02:48:50.552235 26954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:50.551552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.3:57817)
0621 02:48:50.551966 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:50.552059 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":207,"threads_started":1}
I20260621 02:48:50.652230 26955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:50.651608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.1:42029)
0621 02:48:50.652004 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:50.652081 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":209,"threads_started":1}
I20260621 02:48:51.101361 26956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:51.100545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.2:59983)
0621 02:48:51.101089 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:51.101167 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":316,"threads_started":1}
I20260621 02:48:51.553918 26957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:51.553210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.3:51273)
0621 02:48:51.553652 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:51.553759 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":230,"threads_started":1}
I20260621 02:48:51.654075 26958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:51.653195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.1:37791)
0621 02:48:51.653785 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:51.653889 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":297,"threads_started":1}
I20260621 02:48:52.103018 26959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:52.102242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.2:58011)
0621 02:48:52.102733 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:52.102846 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":242,"threads_started":1}
I20260621 02:48:52.555732 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:52.554875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.3:47863)
0621 02:48:52.555363 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:52.555532 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":231,"threads_started":1}
I20260621 02:48:52.655761 26961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:52.654998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.1:35179)
0621 02:48:52.655483 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:52.655567 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":214,"threads_started":1}
I20260621 02:48:53.104789 26962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:53.104006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.2:56047)
0621 02:48:53.104522 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:53.104608 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":282,"threads_started":1}
I20260621 02:48:53.557413 26963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:53.556737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.3:48381)
0621 02:48:53.557165 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:53.557240 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":225,"threads_started":1}
I20260621 02:48:53.657644 26964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:53.656883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.1:54963)
0621 02:48:53.657366 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:53.657452 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":211,"threads_started":1}
I20260621 02:48:54.106757 26965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:54.105958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.2:54229)
0621 02:48:54.106457 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:54.106557 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":276,"threads_started":1}
I20260621 02:48:54.535701 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 26488
I20260621 02:48:54.559224 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:54.558432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:39557 (local address 127.9.166.3:40641)
0621 02:48:54.558937 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:54.559051 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.62:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":284,"threads_started":1}
I20260621 02:48:54.561352 26966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:54.537200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59648 (local address 127.9.166.1:36415)
0621 02:48:54.537895 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:54.537916 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:54.537968 (+ 52us) server_negotiation.cc:407] Connection header received
0621 02:48:54.539755 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:54.539766 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:54.540215 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:54.540520 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:48:54.541951 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:54.543254 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:54.544612 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:54.545342 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:54.545491 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:54.554181 (+ 8690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:48:54.554190 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:48:54.554220 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:48:54.554267 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:48:54.557496 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:54.558358 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:54.558366 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:54.558373 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:54.558556 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:54.558954 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:54.558959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:54.558964 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:54.559641 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:48:54.559972 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:54.560417 (+ 445us) server_negotiation.cc:299] Negotiation successful
0621 02:48:54.561086 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":279,"threads_started":1}
I20260621 02:48:54.562912 26651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:48:54.649060 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 26488
I20260621 02:48:54.663374 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 26659
I20260621 02:48:54.680977 26969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:54.664371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48640 (local address 127.9.166.2:34487)
0621 02:48:54.664894 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:54.664908 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:54.664945 (+ 37us) server_negotiation.cc:407] Connection header received
0621 02:48:54.666415 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:54.666432 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:54.666857 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:54.667223 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:48:54.668685 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:54.669892 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:54.671154 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:54.671892 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:54.672088 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:54.674773 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:48:54.674781 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:48:54.674796 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:48:54.674836 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:48:54.677582 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:54.678281 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:54.678294 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:54.678300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:54.678459 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:54.678857 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:54.678863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:54.678866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:54.679349 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:48:54.679862 (+ 513us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:54.680291 (+ 429us) server_negotiation.cc:299] Negotiation successful
0621 02:48:54.680704 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":245,"threads_started":1}
I20260621 02:48:54.682371 26782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:48:54.764596 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 26659
I20260621 02:48:54.781708 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 26790
I20260621 02:48:54.799481 26971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:54.782665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42940 (local address 127.9.166.3:40409)
0621 02:48:54.783246 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:54.783260 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:54.783308 (+ 48us) server_negotiation.cc:407] Connection header received
0621 02:48:54.784932 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:54.784941 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:54.785367 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:54.785715 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:48:54.787163 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:54.788487 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:54.789697 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:54.790369 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:54.790562 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:54.793227 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:48:54.793234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:48:54.793249 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:48:54.793291 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:48:54.796187 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:54.796896 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:54.796908 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:54.796914 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:54.797073 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:54.797457 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:54.797463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:54.797467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:54.797973 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:48:54.798265 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:54.798707 (+ 442us) server_negotiation.cc:299] Negotiation successful
0621 02:48:54.799125 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":287,"threads_started":1}
I20260621 02:48:54.800933 26913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:48:54.883524 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 26790
2026-06-21T02:48:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64551 ms)
[----------] 30 tests from SecurityITest (290498 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:48:54 dist-test-slave-hvgn krb5kdc[26975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:48:54 dist-test-slave-hvgn krb5kdc[26975](info): set up 2 sockets
Jun 21 02:48:54 dist-test-slave-hvgn krb5kdc[26975](info): commencing operation
krb5kdc: starting...
W20260621 02:48:56.036060 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:48:56 dist-test-slave-hvgn krb5kdc[26975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:48:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:48:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:48:56.270874 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:46121
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:46121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:48:56.595295 26988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:48:56.595685 26988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:48:56.595829 26988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:48:56.603148 26988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:48:56.603241 26988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:48:56.603348 26988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:48:56.603474 26988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:48:56.603559 26988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:48:56.613410 26988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:46121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:46121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:48:56.614734 26988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:48:56.616122 26988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:48:56.627857 26994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:56.627928 26996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:56.627911 26993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:48:56.628948 26988 server_base.cc:1034] running on GCE node
I20260621 02:48:56.629760 26988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:48:56.631242 26988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:48:56.632477 26988 hybrid_clock.cc:648] HybridClock initialized: now 1782010136632420 us; error 75 us; skew 500 ppm
I20260621 02:48:56.632829 26988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:48:56 dist-test-slave-hvgn krb5kdc[26975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010136, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:48:56.635900 26988 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:48:56.637501 26988 webserver.cc:460] Webserver started at http://127.9.166.62:34307/ using document root <none> and password file <none>
I20260621 02:48:56.638199 26988 fs_manager.cc:359] Metadata directory not provided
I20260621 02:48:56.638351 26988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:48:56.638661 26988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:48:56.641799 26988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "80e74bb8adc44f0b8b2ca4d07eadd22b"
format_stamp: "Formatted at 2026-06-21 02:48:56 on dist-test-slave-hvgn"
server_key: "13a75af93defaadeb94e9bcd0d0d8d3c"
server_key_iv: "6048a723eae5782bfed95f8203469493"
server_key_version: "encryptionkey@0"
I20260621 02:48:56.642521 26988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "80e74bb8adc44f0b8b2ca4d07eadd22b"
format_stamp: "Formatted at 2026-06-21 02:48:56 on dist-test-slave-hvgn"
server_key: "13a75af93defaadeb94e9bcd0d0d8d3c"
server_key_iv: "6048a723eae5782bfed95f8203469493"
server_key_version: "encryptionkey@0"
I20260621 02:48:56.647534 26988 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 02:48:56.650734 27003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:56.652048 26988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:48:56.652235 26988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "80e74bb8adc44f0b8b2ca4d07eadd22b"
format_stamp: "Formatted at 2026-06-21 02:48:56 on dist-test-slave-hvgn"
server_key: "13a75af93defaadeb94e9bcd0d0d8d3c"
server_key_iv: "6048a723eae5782bfed95f8203469493"
server_key_version: "encryptionkey@0"
I20260621 02:48:56.652441 26988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:48:56.675926 26988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:48:56.679826 26988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:48:56.680104 26988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:48:56.698802 26988 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:46121
I20260621 02:48:56.698840 27054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:46121 every 8 connection(s)
I20260621 02:48:56.700162 26988 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:48:56.701597 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 26988
I20260621 02:48:56.701828 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:48:56.702483 9880 external_mini_cluster.cc:1428] Setting key 398d70d317c580f49364b1e72727a716
I20260621 02:48:56.704356 27055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:48:56 dist-test-slave-hvgn krb5kdc[26975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:48:56.718122 27055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b: Bootstrap starting.
I20260621 02:48:56.721588 27055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b: Neither blocks nor log segments found. Creating new log.
I20260621 02:48:56.722591 27055 log.cc:824] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b: Log is configured to *not* fsync() on all Append() calls
I20260621 02:48:56.725525 27055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b: No bootstrap required, opened a new log
I20260621 02:48:56.725560 27058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:56.704449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47478 (local address 127.9.166.62:46121)
0621 02:48:56.705270 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:56.705282 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:56.705322 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:48:56.706938 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:56.706959 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:56.707348 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:56.707750 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:48:56.709431 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:56.710728 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:56.712621 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:56.713314 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:56.713442 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:56.718401 (+ 4959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:48:56.718409 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:48:56.718422 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:48:56.718457 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:48:56.721476 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:56.722246 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:56.722252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:56.722258 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:56.722424 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:56.722830 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:56.722834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:56.722836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:56.723471 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:48:56.723784 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:56.724201 (+ 417us) server_negotiation.cc:299] Negotiation successful
0621 02:48:56.724630 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":382,"threads_started":1}
I20260621 02:48:56.733762 27055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80e74bb8adc44f0b8b2ca4d07eadd22b" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46121 } }
I20260621 02:48:56.734171 27055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:48:56.734278 27055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80e74bb8adc44f0b8b2ca4d07eadd22b, State: Initialized, Role: FOLLOWER
I20260621 02:48:56.734829 27055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80e74bb8adc44f0b8b2ca4d07eadd22b" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46121 } }
I20260621 02:48:56.735067 27055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:48:56.735212 27055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:48:56.735406 27055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:48:56.737610 27055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80e74bb8adc44f0b8b2ca4d07eadd22b" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46121 } }
I20260621 02:48:56.738099 27055 leader_election.cc:302] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80e74bb8adc44f0b8b2ca4d07eadd22b; no voters:
I20260621 02:48:56.738744 27055 leader_election.cc:288] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:48:56.738903 27060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:48:56.739636 27060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b [term 1 LEADER]: Becoming Leader. State: Replica: 80e74bb8adc44f0b8b2ca4d07eadd22b, State: Running, Role: LEADER
I20260621 02:48:56.740238 27060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80e74bb8adc44f0b8b2ca4d07eadd22b" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46121 } }
I20260621 02:48:56.740818 27055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:48:56.743237 27061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80e74bb8adc44f0b8b2ca4d07eadd22b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80e74bb8adc44f0b8b2ca4d07eadd22b" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46121 } } }
I20260621 02:48:56.744433 27061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b [sys.catalog]: This master's current role is: LEADER
I20260621 02:48:56.744212 27062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80e74bb8adc44f0b8b2ca4d07eadd22b. Latest consensus state: current_term: 1 leader_uuid: "80e74bb8adc44f0b8b2ca4d07eadd22b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80e74bb8adc44f0b8b2ca4d07eadd22b" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46121 } } }
I20260621 02:48:56.745077 27062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b [sys.catalog]: This master's current role is: LEADER
W20260621 02:48:56.754673 27076 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:48:56.754868 27076 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:48:56.755055 27069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:48:56.757222 27069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:48:56.766105 27069 catalog_manager.cc:1347] Generated new cluster ID: d82e222ff1884821b4fc6f30161cd091
I20260621 02:48:56.766224 27069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:48:56.776564 27069 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:48:56.777769 27069 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:48:56.788592 27069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b: Generated new TSK 0
I20260621 02:48:56.789354 27069 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
I20260621 02:48:56.848449 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:46121
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:48:57.176867 27081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:48:57.177137 27081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:48:57.177295 27081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:48:57.184373 27081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:48:57.184492 27081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:48:57.184681 27081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:48:57.194015 27081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:46121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:48:57.195216 27081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:48:57.196517 27081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:48:57.209678 27086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:57.209684 27089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:57.209714 27087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:48:57.210502 27081 server_base.cc:1034] running on GCE node
I20260621 02:48:57.211130 27081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:48:57.212107 27081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:48:57.213315 27081 hybrid_clock.cc:648] HybridClock initialized: now 1782010137213265 us; error 45 us; skew 500 ppm
I20260621 02:48:57.213636 27081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:48:57 dist-test-slave-hvgn krb5kdc[26975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010137, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:48:57.216820 27081 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:48:57.218318 27081 webserver.cc:460] Webserver started at http://127.9.166.1:35729/ using document root <none> and password file <none>
I20260621 02:48:57.219018 27081 fs_manager.cc:359] Metadata directory not provided
I20260621 02:48:57.219163 27081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:48:57.219533 27081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:48:57.222430 27081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "d47e0ed11dce4aefb5b6de6db220ba94"
format_stamp: "Formatted at 2026-06-21 02:48:57 on dist-test-slave-hvgn"
server_key: "f0b544e17502ff353edb347fc4489bba"
server_key_iv: "96da81ea12a99c21485cb48868d33688"
server_key_version: "encryptionkey@0"
I20260621 02:48:57.223223 27081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "d47e0ed11dce4aefb5b6de6db220ba94"
format_stamp: "Formatted at 2026-06-21 02:48:57 on dist-test-slave-hvgn"
server_key: "f0b544e17502ff353edb347fc4489bba"
server_key_iv: "96da81ea12a99c21485cb48868d33688"
server_key_version: "encryptionkey@0"
I20260621 02:48:57.228368 27081 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 02:48:57.231797 27096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:57.233189 27081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:48:57.233384 27081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "d47e0ed11dce4aefb5b6de6db220ba94"
format_stamp: "Formatted at 2026-06-21 02:48:57 on dist-test-slave-hvgn"
server_key: "f0b544e17502ff353edb347fc4489bba"
server_key_iv: "96da81ea12a99c21485cb48868d33688"
server_key_version: "encryptionkey@0"
I20260621 02:48:57.233582 27081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:48:57.250805 27081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:48:57.254556 27081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:48:57.254845 27081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:48:57.255869 27081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:48:57.257251 27081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:48:57.257352 27081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:57.257501 27081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:48:57.257573 27081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:57.295240 27081 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:36547
I20260621 02:48:57.295323 27208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:36547 every 8 connection(s)
I20260621 02:48:57.296559 27081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:48:57.302259 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 27081
I20260621 02:48:57.302475 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:48:57.303162 9880 external_mini_cluster.cc:1428] Setting key da9f6ecb5f28d51f14f11e55ee62b190
Jun 21 02:48:57 dist-test-slave-hvgn krb5kdc[26975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010137, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:48:57.316170 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:57.299078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:36167 (local address 127.9.166.62:46121)
0621 02:48:57.299589 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:57.299595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:57.300545 (+ 950us) server_negotiation.cc:407] Connection header received
0621 02:48:57.302311 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:57.302317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:57.302421 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:57.302554 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:48:57.304216 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:57.304961 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:57.306470 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:57.306867 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:57.306999 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:57.310439 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:48:57.310444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:48:57.310447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:48:57.310479 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:48:57.313077 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:57.313845 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:57.313855 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:57.313858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:57.313975 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:57.314411 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:57.314416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:57.314419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:57.314681 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:48:57.314831 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:57.315554 (+ 723us) server_negotiation.cc:299] Negotiation successful
0621 02:48:57.315796 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":236,"threads_started":1}
I20260621 02:48:57.317116 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:57.299690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:46121 (local address 127.9.166.1:36167)
0621 02:48:57.300384 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:57.300426 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0621 02:48:57.302048 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:48:57.302789 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:48:57.302796 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:48:57.303330 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:48:57.304028 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:57.304056 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:57.305110 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:57.305115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:57.306316 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:57.306327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:57.307091 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:57.307100 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:57.307329 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:57.308047 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:48:57.308068 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:48:57.310196 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:48:57.313232 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:48:57.313241 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:48:57.313268 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:48:57.313714 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:48:57.314086 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:48:57.314091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:48:57.314095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:48:57.314294 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:48:57.314816 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:48:57.314823 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:48:57.315166 (+ 343us) client_negotiation.cc:769] Sending connection context
0621 02:48:57.315445 (+ 279us) client_negotiation.cc:240] Negotiation successful
0621 02:48:57.315976 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":277,"threads_started":1}
I20260621 02:48:57.318696 27209 heartbeater.cc:347] Connected to a master server at 127.9.166.62:46121
I20260621 02:48:57.319010 27209 heartbeater.cc:464] Registering TS with master...
I20260621 02:48:57.319763 27209 heartbeater.cc:511] Master 127.9.166.62:46121 requested a full tablet report, sending...
I20260621 02:48:57.322089 27020 ts_manager.cc:194] Registered new tserver with Master: d47e0ed11dce4aefb5b6de6db220ba94 (127.9.166.1:36547)
I20260621 02:48:57.324386 27020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:36167
WARNING: no policy specified for kudu/127.9.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
I20260621 02:48:57.356912 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:46121
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:48:57.679095 27215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:48:57.679358 27215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:48:57.679512 27215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:48:57.686522 27215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:48:57.686625 27215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:48:57.686753 27215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:48:57.696157 27215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:46121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:48:57.697294 27215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:48:57.698801 27215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:48:57.713260 27220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:57.713234 27221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:57.714196 27223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:48:57.714308 27215 server_base.cc:1034] running on GCE node
I20260621 02:48:57.714917 27215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:48:57.715855 27215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:48:57.717085 27215 hybrid_clock.cc:648] HybridClock initialized: now 1782010137717032 us; error 62 us; skew 500 ppm
I20260621 02:48:57.717399 27215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:48:57 dist-test-slave-hvgn krb5kdc[26975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010137, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:48:57.720496 27215 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:48:57.721877 27215 webserver.cc:460] Webserver started at http://127.9.166.2:36439/ using document root <none> and password file <none>
I20260621 02:48:57.722564 27215 fs_manager.cc:359] Metadata directory not provided
I20260621 02:48:57.722712 27215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:48:57.723016 27215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:48:57.725934 27215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "bcd71a5e6d2b48acbd3b75bedd19d07c"
format_stamp: "Formatted at 2026-06-21 02:48:57 on dist-test-slave-hvgn"
server_key: "08e448946d9785f2777360c5afd29b1a"
server_key_iv: "77e104bdc1fe9658a06e91ef84744b2d"
server_key_version: "encryptionkey@0"
I20260621 02:48:57.726691 27215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "bcd71a5e6d2b48acbd3b75bedd19d07c"
format_stamp: "Formatted at 2026-06-21 02:48:57 on dist-test-slave-hvgn"
server_key: "08e448946d9785f2777360c5afd29b1a"
server_key_iv: "77e104bdc1fe9658a06e91ef84744b2d"
server_key_version: "encryptionkey@0"
I20260621 02:48:57.731724 27215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260621 02:48:57.735065 27230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:57.736550 27215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:48:57.736708 27215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "bcd71a5e6d2b48acbd3b75bedd19d07c"
format_stamp: "Formatted at 2026-06-21 02:48:57 on dist-test-slave-hvgn"
server_key: "08e448946d9785f2777360c5afd29b1a"
server_key_iv: "77e104bdc1fe9658a06e91ef84744b2d"
server_key_version: "encryptionkey@0"
I20260621 02:48:57.736946 27215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:48:57.752214 27215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:48:57.756157 27215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:48:57.756426 27215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:48:57.757382 27215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:48:57.758636 27215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:48:57.758736 27215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:57.758882 27215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:48:57.758956 27215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:57.790781 27215 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:39773
I20260621 02:48:57.790823 27342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:39773 every 8 connection(s)
I20260621 02:48:57.792150 27215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:48:57.799973 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 27215
I20260621 02:48:57.800164 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:48:57.800791 9880 external_mini_cluster.cc:1428] Setting key 22ce62be47bdafd85d594aef85f8b130
Jun 21 02:48:57 dist-test-slave-hvgn krb5kdc[26975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010137, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:48:57.813243 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:57.794700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:55879 (local address 127.9.166.62:46121)
0621 02:48:57.794904 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:57.794912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:57.796172 (+ 1260us) server_negotiation.cc:407] Connection header received
0621 02:48:57.797866 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:57.797871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:57.797937 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:57.798025 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:48:57.799815 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:57.800994 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:57.802540 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:57.802980 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:57.803123 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:57.806911 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:48:57.806916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:48:57.806919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:48:57.806950 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:48:57.809810 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:57.810656 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:57.810666 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:57.810669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:57.810797 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:57.811272 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:57.811282 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:57.811284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:57.811647 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:48:57.811848 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:57.812612 (+ 764us) server_negotiation.cc:299] Negotiation successful
0621 02:48:57.812863 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:48:57.814379 27345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:57.795237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:46121 (local address 127.9.166.2:55879)
0621 02:48:57.795989 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:57.796026 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0621 02:48:57.797628 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:48:57.798348 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:48:57.798355 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:48:57.798985 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:48:57.799628 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:57.799648 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:57.801182 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:57.801186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:57.802389 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:57.802401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:57.803216 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:57.803223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:57.803517 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:57.804277 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:48:57.804306 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:48:57.806681 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:48:57.809980 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:48:57.809990 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:48:57.810009 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:48:57.810487 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:48:57.810922 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:48:57.810928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:48:57.810932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:48:57.811139 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:48:57.811800 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:48:57.811813 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:48:57.812203 (+ 390us) client_negotiation.cc:769] Sending connection context
0621 02:48:57.812496 (+ 293us) client_negotiation.cc:240] Negotiation successful
0621 02:48:57.813094 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":351,"threads_started":1}
I20260621 02:48:57.816069 27343 heartbeater.cc:347] Connected to a master server at 127.9.166.62:46121
I20260621 02:48:57.816393 27343 heartbeater.cc:464] Registering TS with master...
I20260621 02:48:57.817145 27343 heartbeater.cc:511] Master 127.9.166.62:46121 requested a full tablet report, sending...
I20260621 02:48:57.818809 27020 ts_manager.cc:194] Registered new tserver with Master: bcd71a5e6d2b48acbd3b75bedd19d07c (127.9.166.2:39773)
I20260621 02:48:57.820159 27020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:55879
WARNING: no policy specified for kudu/127.9.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
I20260621 02:48:57.854878 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:46121
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:48:58.173096 27348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:48:58.173353 27348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:48:58.173593 27348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:48:58.180989 27348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:48:58.181084 27348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:48:58.181278 27348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:48:58.190615 27348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:46121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:48:58.191847 27348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:48:58.193089 27348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:48:58.205864 27353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:58.205940 27354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:58.206183 27356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:48:58.206950 27348 server_base.cc:1034] running on GCE node
I20260621 02:48:58.207620 27348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:48:58.208504 27348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:48:58.209736 27348 hybrid_clock.cc:648] HybridClock initialized: now 1782010138209679 us; error 38 us; skew 500 ppm
I20260621 02:48:58.210104 27348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:48:58 dist-test-slave-hvgn krb5kdc[26975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010138, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:48:58.213490 27348 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:48:58.214923 27348 webserver.cc:460] Webserver started at http://127.9.166.3:39445/ using document root <none> and password file <none>
I20260621 02:48:58.215682 27348 fs_manager.cc:359] Metadata directory not provided
I20260621 02:48:58.215813 27348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:48:58.216125 27348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:48:58.218933 27348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "356da5c18f9b4cd8810eda1841472eb6"
format_stamp: "Formatted at 2026-06-21 02:48:58 on dist-test-slave-hvgn"
server_key: "0656603861be382f70a3535a504cce91"
server_key_iv: "f21c3e6ee21462940af483ef1599f1af"
server_key_version: "encryptionkey@0"
I20260621 02:48:58.219756 27348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "356da5c18f9b4cd8810eda1841472eb6"
format_stamp: "Formatted at 2026-06-21 02:48:58 on dist-test-slave-hvgn"
server_key: "0656603861be382f70a3535a504cce91"
server_key_iv: "f21c3e6ee21462940af483ef1599f1af"
server_key_version: "encryptionkey@0"
I20260621 02:48:58.224660 27348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260621 02:48:58.228298 27363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:58.229632 27348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:48:58.229802 27348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "356da5c18f9b4cd8810eda1841472eb6"
format_stamp: "Formatted at 2026-06-21 02:48:58 on dist-test-slave-hvgn"
server_key: "0656603861be382f70a3535a504cce91"
server_key_iv: "f21c3e6ee21462940af483ef1599f1af"
server_key_version: "encryptionkey@0"
I20260621 02:48:58.229991 27348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:48:58.254594 27348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:48:58.259181 27348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:48:58.259604 27348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:48:58.260627 27348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:48:58.261931 27348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:48:58.262005 27348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:58.262179 27348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:48:58.262230 27348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:58.294687 27348 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:41919
I20260621 02:48:58.294790 27475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:41919 every 8 connection(s)
I20260621 02:48:58.296023 27348 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:48:58.298676 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 27348
I20260621 02:48:58.298875 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:48:58.299352 9880 external_mini_cluster.cc:1428] Setting key 2c7c4a124b9412055a8979707a66e4bb
Jun 21 02:48:58 dist-test-slave-hvgn krb5kdc[26975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010138, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:48:58.316387 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:58.299874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:57325 (local address 127.9.166.62:46121)
0621 02:48:58.300086 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:58.300095 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:58.300435 (+ 340us) server_negotiation.cc:407] Connection header received
0621 02:48:58.302857 (+ 2422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:58.302860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:58.302954 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:58.303061 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:48:58.304471 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.305200 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.306699 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.307098 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.307191 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:58.310572 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:48:58.310577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:48:58.310580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:48:58.310611 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:48:58.313137 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:58.313981 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:58.313991 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:58.313993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:58.314130 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:58.314571 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:58.314575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:58.314577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:58.314842 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:48:58.314999 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:58.315813 (+ 814us) server_negotiation.cc:299] Negotiation successful
0621 02:48:58.316035 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:48:58.317334 27478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:58.299319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:46121 (local address 127.9.166.3:57325)
0621 02:48:58.300263 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:58.300312 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0621 02:48:58.302583 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:48:58.303223 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:48:58.303230 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:48:58.303795 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:48:58.304335 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:58.304350 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.305325 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.305329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:58.306582 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:58.306594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.307286 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.307290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:58.307561 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:58.308251 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:48:58.308272 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:48:58.310396 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:48:58.313269 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:48:58.313277 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:48:58.313292 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:48:58.313859 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:48:58.314241 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:48:58.314247 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:48:58.314251 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:48:58.314446 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:48:58.314976 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:48:58.314983 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:48:58.315370 (+ 387us) client_negotiation.cc:769] Sending connection context
0621 02:48:58.315698 (+ 328us) client_negotiation.cc:240] Negotiation successful
0621 02:48:58.316187 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":387,"threads_started":1}
I20260621 02:48:58.318619 27476 heartbeater.cc:347] Connected to a master server at 127.9.166.62:46121
I20260621 02:48:58.318938 27476 heartbeater.cc:464] Registering TS with master...
I20260621 02:48:58.319689 27476 heartbeater.cc:511] Master 127.9.166.62:46121 requested a full tablet report, sending...
I20260621 02:48:58.321077 27020 ts_manager.cc:194] Registered new tserver with Master: 356da5c18f9b4cd8810eda1841472eb6 (127.9.166.3:41919)
I20260621 02:48:58.321856 27020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:57325
I20260621 02:48:58.327296 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:48:58.327523 27209 heartbeater.cc:503] Master 127.9.166.62:46121 was elected leader, sending a full tablet report...
I20260621 02:48:58.344467 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:58.332026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.9.166.62:46121)
0621 02:48:58.332192 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:58.332196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:58.332731 (+ 535us) server_negotiation.cc:407] Connection header received
0621 02:48:58.332889 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:58.332897 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:58.332952 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:58.333030 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:48:58.334400 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.335175 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.336494 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.336888 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.336979 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:58.339372 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:48:58.339376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:48:58.339380 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:48:58.339450 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:48:58.342032 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:58.342710 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:58.342714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:58.342716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:58.342844 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:58.343182 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:58.343185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:58.343187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:58.343522 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:48:58.343708 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:58.343988 (+ 280us) server_negotiation.cc:299] Negotiation successful
0621 02:48:58.344182 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 02:48:58.348663 27020 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:48:58.355515 27020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:48:58.379789 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:58.367142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:36547 (local address 127.0.0.1:44904)
0621 02:48:58.368031 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:58.368072 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 02:48:58.368251 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:48:58.369657 (+ 1406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:48:58.369668 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:48:58.369726 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:48:58.370217 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:58.370233 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.373210 (+ 2977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.373219 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:58.375998 (+ 2779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:58.376018 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.378520 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.378542 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:58.378844 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:58.378922 (+ 78us) client_negotiation.cc:769] Sending connection context
0621 02:48:58.379052 (+ 130us) client_negotiation.cc:240] Negotiation successful
0621 02:48:58.379242 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"spinlock_wait_cycles":1280,"thread_start_us":2214,"threads_started":1}
I20260621 02:48:58.381253 27489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:58.367301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44904 (local address 127.9.166.1:36547)
0621 02:48:58.368910 (+ 1609us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:58.368936 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:58.368979 (+ 43us) server_negotiation.cc:407] Connection header received
0621 02:48:58.369101 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:58.369112 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:58.369374 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:58.370462 (+ 1088us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:48:58.370742 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.373028 (+ 2286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.376259 (+ 3231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.378259 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.380452 (+ 2193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:58.380698 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:58.380762 (+ 64us) server_negotiation.cc:299] Negotiation successful
0621 02:48:58.380928 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1430,"thread_start_us":1288,"threads_started":1}
I20260621 02:48:58.385010 27144 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4cb82024d44f54b23e1a1401152bc4 (DEFAULT_TABLE table=test-table [id=b02de2954a324fe3bb59b9409b6fc294]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:48:58.386384 27144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4cb82024d44f54b23e1a1401152bc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:48:58.388355 27491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:58.367912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40036 (local address 127.9.166.3:41919)
0621 02:48:58.381046 (+ 13134us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:58.381055 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:58.381087 (+ 32us) server_negotiation.cc:407] Connection header received
0621 02:48:58.381184 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:58.381193 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:58.381438 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:58.381640 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:48:58.382216 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.383724 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.386246 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.387265 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.387402 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:58.387659 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:58.387870 (+ 211us) server_negotiation.cc:299] Negotiation successful
0621 02:48:58.388021 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12971,"thread_start_us":12816,"threads_started":1}
I20260621 02:48:58.388396 27488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:58.367245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:41919 (local address 127.0.0.1:40036)
0621 02:48:58.369592 (+ 2347us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:58.369611 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:48:58.369744 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:48:58.381644 (+ 11900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:48:58.381649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:48:58.381675 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:48:58.382023 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:58.382036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.383871 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.383877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:58.386105 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:58.386115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.387484 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.387490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:58.387656 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:58.387695 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 02:48:58.387828 (+ 133us) client_negotiation.cc:240] Negotiation successful
0621 02:48:58.387956 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2233,"spinlock_wait_cycles":11904,"thread_start_us":917,"threads_started":1}
I20260621 02:48:58.388963 27493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:58.370227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.9.166.2:39773)
0621 02:48:58.380429 (+ 10202us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:58.380437 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:58.380464 (+ 27us) server_negotiation.cc:407] Connection header received
0621 02:48:58.380565 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:58.380573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:58.380824 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:58.381035 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:48:58.381591 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.383216 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.385894 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.387313 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.387465 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:58.387660 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:58.388490 (+ 830us) server_negotiation.cc:299] Negotiation successful
0621 02:48:58.388651 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10045,"thread_start_us":9911,"threads_started":1}
I20260621 02:48:58.389503 27492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:58.369742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:39773 (local address 127.0.0.1:57238)
0621 02:48:58.379565 (+ 9823us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:58.379588 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:48:58.379741 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:48:58.381041 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:48:58.381046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:48:58.381075 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:48:58.381372 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:58.381383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.383364 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.383368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:58.385721 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:58.385736 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.387809 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.387816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:58.387982 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:58.388012 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 02:48:58.388106 (+ 94us) client_negotiation.cc:240] Negotiation successful
0621 02:48:58.388240 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9619,"thread_start_us":10111,"threads_started":1}
I20260621 02:48:58.391968 27411 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4cb82024d44f54b23e1a1401152bc4 (DEFAULT_TABLE table=test-table [id=b02de2954a324fe3bb59b9409b6fc294]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:48:58.393096 27411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4cb82024d44f54b23e1a1401152bc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:48:58.392899 27278 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4cb82024d44f54b23e1a1401152bc4 (DEFAULT_TABLE table=test-table [id=b02de2954a324fe3bb59b9409b6fc294]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:48:58.394142 27278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4cb82024d44f54b23e1a1401152bc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:48:58.401305 27494 tablet_bootstrap.cc:492] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94: Bootstrap starting.
I20260621 02:48:58.404989 27494 tablet_bootstrap.cc:654] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94: Neither blocks nor log segments found. Creating new log.
I20260621 02:48:58.405805 27495 tablet_bootstrap.cc:492] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6: Bootstrap starting.
I20260621 02:48:58.407642 27496 tablet_bootstrap.cc:492] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c: Bootstrap starting.
I20260621 02:48:58.407742 27494 log.cc:824] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94: Log is configured to *not* fsync() on all Append() calls
I20260621 02:48:58.409365 27495 tablet_bootstrap.cc:654] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6: Neither blocks nor log segments found. Creating new log.
I20260621 02:48:58.410315 27495 log.cc:824] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6: Log is configured to *not* fsync() on all Append() calls
I20260621 02:48:58.410846 27496 tablet_bootstrap.cc:654] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c: Neither blocks nor log segments found. Creating new log.
I20260621 02:48:58.411237 27494 tablet_bootstrap.cc:492] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94: No bootstrap required, opened a new log
I20260621 02:48:58.411538 27494 ts_tablet_manager.cc:1397] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260621 02:48:58.412074 27496 log.cc:824] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:48:58.413097 27495 tablet_bootstrap.cc:492] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6: No bootstrap required, opened a new log
I20260621 02:48:58.413358 27495 ts_tablet_manager.cc:1397] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260621 02:48:58.414832 27496 tablet_bootstrap.cc:492] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c: No bootstrap required, opened a new log
I20260621 02:48:58.415081 27496 ts_tablet_manager.cc:1397] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260621 02:48:58.420666 27495 raft_consensus.cc:348] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47e0ed11dce4aefb5b6de6db220ba94" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36547 } } peers { permanent_uuid: "356da5c18f9b4cd8810eda1841472eb6" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41919 } } peers { permanent_uuid: "bcd71a5e6d2b48acbd3b75bedd19d07c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39773 } }
I20260621 02:48:58.421021 27495 raft_consensus.cc:374] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:48:58.421120 27495 raft_consensus.cc:729] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 356da5c18f9b4cd8810eda1841472eb6, State: Initialized, Role: FOLLOWER
I20260621 02:48:58.421737 27495 consensus_queue.cc:260] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47e0ed11dce4aefb5b6de6db220ba94" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36547 } } peers { permanent_uuid: "356da5c18f9b4cd8810eda1841472eb6" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41919 } } peers { permanent_uuid: "bcd71a5e6d2b48acbd3b75bedd19d07c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39773 } }
I20260621 02:48:58.422487 27494 raft_consensus.cc:348] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47e0ed11dce4aefb5b6de6db220ba94" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36547 } } peers { permanent_uuid: "356da5c18f9b4cd8810eda1841472eb6" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41919 } } peers { permanent_uuid: "bcd71a5e6d2b48acbd3b75bedd19d07c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39773 } }
I20260621 02:48:58.422772 27476 heartbeater.cc:503] Master 127.9.166.62:46121 was elected leader, sending a full tablet report...
I20260621 02:48:58.422912 27494 raft_consensus.cc:374] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:48:58.423010 27494 raft_consensus.cc:729] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d47e0ed11dce4aefb5b6de6db220ba94, State: Initialized, Role: FOLLOWER
I20260621 02:48:58.423542 27495 ts_tablet_manager.cc:1428] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260621 02:48:58.423575 27494 consensus_queue.cc:260] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47e0ed11dce4aefb5b6de6db220ba94" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36547 } } peers { permanent_uuid: "356da5c18f9b4cd8810eda1841472eb6" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41919 } } peers { permanent_uuid: "bcd71a5e6d2b48acbd3b75bedd19d07c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39773 } }
I20260621 02:48:58.425263 27494 ts_tablet_manager.cc:1428] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260621 02:48:58.426083 27496 raft_consensus.cc:348] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47e0ed11dce4aefb5b6de6db220ba94" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36547 } } peers { permanent_uuid: "356da5c18f9b4cd8810eda1841472eb6" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41919 } } peers { permanent_uuid: "bcd71a5e6d2b48acbd3b75bedd19d07c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39773 } }
I20260621 02:48:58.426512 27496 raft_consensus.cc:374] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:48:58.426617 27496 raft_consensus.cc:729] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcd71a5e6d2b48acbd3b75bedd19d07c, State: Initialized, Role: FOLLOWER
I20260621 02:48:58.427199 27496 consensus_queue.cc:260] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47e0ed11dce4aefb5b6de6db220ba94" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36547 } } peers { permanent_uuid: "356da5c18f9b4cd8810eda1841472eb6" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41919 } } peers { permanent_uuid: "bcd71a5e6d2b48acbd3b75bedd19d07c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39773 } }
I20260621 02:48:58.428391 27343 heartbeater.cc:503] Master 127.9.166.62:46121 was elected leader, sending a full tablet report...
I20260621 02:48:58.429090 27496 ts_tablet_manager.cc:1428] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
W20260621 02:48:58.544255 27344 tablet.cc:2369] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:48:58.547700 27477 tablet.cc:2369] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:48:58.549019 27210 tablet.cc:2369] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:48:58.618968 27502 raft_consensus.cc:482] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:48:58.619275 27502 raft_consensus.cc:504] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47e0ed11dce4aefb5b6de6db220ba94" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36547 } } peers { permanent_uuid: "356da5c18f9b4cd8810eda1841472eb6" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41919 } } peers { permanent_uuid: "bcd71a5e6d2b48acbd3b75bedd19d07c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39773 } }
I20260621 02:48:58.620967 27502 leader_election.cc:288] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d47e0ed11dce4aefb5b6de6db220ba94 (127.9.166.1:36547), 356da5c18f9b4cd8810eda1841472eb6 (127.9.166.3:41919)
I20260621 02:48:58.628579 27491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:58.621734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:50617 (local address 127.9.166.3:41919)
0621 02:48:58.621975 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:58.621980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:58.622565 (+ 585us) server_negotiation.cc:407] Connection header received
0621 02:48:58.622734 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:58.622743 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:58.622804 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:58.622871 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:48:58.623641 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.624450 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.626301 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.627562 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.627704 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:58.627810 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:58.628142 (+ 332us) server_negotiation.cc:299] Negotiation successful
0621 02:48:58.628271 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260621 02:48:58.628611 27489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:58.621463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:47515 (local address 127.9.166.1:36547)
0621 02:48:58.621684 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:58.621693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:58.622497 (+ 804us) server_negotiation.cc:407] Connection header received
0621 02:48:58.622782 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:58.622786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:58.622867 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:58.622957 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:48:58.623735 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.624434 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.626473 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.627743 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.627873 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:58.627943 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:58.628273 (+ 330us) server_negotiation.cc:299] Negotiation successful
0621 02:48:58.628399 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:48:58.628671 27503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:58.621424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:36547 (local address 127.9.166.2:47515)
0621 02:48:58.622361 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:58.622386 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0621 02:48:58.622583 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:48:58.622991 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:48:58.622998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:48:58.623027 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:48:58.623589 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:58.623605 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.624575 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.624581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:58.626317 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:58.626331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.627940 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.627946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:58.628130 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:58.628157 (+ 27us) client_negotiation.cc:769] Sending connection context
0621 02:48:58.628233 (+ 76us) client_negotiation.cc:240] Negotiation successful
0621 02:48:58.628327 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":70,"thread_start_us":747,"threads_started":1}
I20260621 02:48:58.628777 27504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:58.621441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:41919 (local address 127.9.166.2:50617)
0621 02:48:58.622476 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0621 02:48:58.622491 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:48:58.622597 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:48:58.622919 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:48:58.622925 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:48:58.622952 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:48:58.623435 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:58.623469 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.624611 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.624616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:58.626143 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:48:58.626154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.627763 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.627767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:48:58.627964 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:58.628009 (+ 45us) client_negotiation.cc:769] Sending connection context
0621 02:48:58.628109 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 02:48:58.628246 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"mutex_wait_us":1,"spinlock_wait_cycles":23808,"thread_start_us":498,"threads_started":1}
I20260621 02:48:58.629786 27164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4cb82024d44f54b23e1a1401152bc4" candidate_uuid: "bcd71a5e6d2b48acbd3b75bedd19d07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d47e0ed11dce4aefb5b6de6db220ba94" is_pre_election: true
I20260621 02:48:58.629962 27431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4cb82024d44f54b23e1a1401152bc4" candidate_uuid: "bcd71a5e6d2b48acbd3b75bedd19d07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "356da5c18f9b4cd8810eda1841472eb6" is_pre_election: true
I20260621 02:48:58.630388 27164 raft_consensus.cc:2454] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcd71a5e6d2b48acbd3b75bedd19d07c in term 0.
I20260621 02:48:58.630472 27431 raft_consensus.cc:2454] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcd71a5e6d2b48acbd3b75bedd19d07c in term 0.
I20260621 02:48:58.631168 27233 leader_election.cc:302] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 356da5c18f9b4cd8810eda1841472eb6, bcd71a5e6d2b48acbd3b75bedd19d07c, d47e0ed11dce4aefb5b6de6db220ba94; no voters:
I20260621 02:48:58.631577 27502 raft_consensus.cc:2781] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:48:58.631757 27502 raft_consensus.cc:482] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:48:58.631860 27502 raft_consensus.cc:3037] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:48:58.634085 27502 raft_consensus.cc:504] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47e0ed11dce4aefb5b6de6db220ba94" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36547 } } peers { permanent_uuid: "356da5c18f9b4cd8810eda1841472eb6" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41919 } } peers { permanent_uuid: "bcd71a5e6d2b48acbd3b75bedd19d07c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39773 } }
I20260621 02:48:58.634712 27502 leader_election.cc:288] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [CANDIDATE]: Term 1 election: Requested vote from peers d47e0ed11dce4aefb5b6de6db220ba94 (127.9.166.1:36547), 356da5c18f9b4cd8810eda1841472eb6 (127.9.166.3:41919)
I20260621 02:48:58.635123 27164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4cb82024d44f54b23e1a1401152bc4" candidate_uuid: "bcd71a5e6d2b48acbd3b75bedd19d07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d47e0ed11dce4aefb5b6de6db220ba94"
I20260621 02:48:58.635326 27164 raft_consensus.cc:3037] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:48:58.635435 27431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4cb82024d44f54b23e1a1401152bc4" candidate_uuid: "bcd71a5e6d2b48acbd3b75bedd19d07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "356da5c18f9b4cd8810eda1841472eb6"
I20260621 02:48:58.635638 27431 raft_consensus.cc:3037] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:48:58.638688 27164 raft_consensus.cc:2454] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcd71a5e6d2b48acbd3b75bedd19d07c in term 1.
I20260621 02:48:58.638728 27431 raft_consensus.cc:2454] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcd71a5e6d2b48acbd3b75bedd19d07c in term 1.
I20260621 02:48:58.639083 27233 leader_election.cc:302] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcd71a5e6d2b48acbd3b75bedd19d07c, d47e0ed11dce4aefb5b6de6db220ba94; no voters:
I20260621 02:48:58.639339 27502 raft_consensus.cc:2781] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:48:58.639837 27502 raft_consensus.cc:686] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [term 1 LEADER]: Becoming Leader. State: Replica: bcd71a5e6d2b48acbd3b75bedd19d07c, State: Running, Role: LEADER
I20260621 02:48:58.640410 27502 consensus_queue.cc:237] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47e0ed11dce4aefb5b6de6db220ba94" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36547 } } peers { permanent_uuid: "356da5c18f9b4cd8810eda1841472eb6" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41919 } } peers { permanent_uuid: "bcd71a5e6d2b48acbd3b75bedd19d07c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39773 } }
I20260621 02:48:58.645365 27019 catalog_manager.cc:5495] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c reported cstate change: term changed from 0 to 1, leader changed from <none> to bcd71a5e6d2b48acbd3b75bedd19d07c (127.9.166.2). New cstate: current_term: 1 leader_uuid: "bcd71a5e6d2b48acbd3b75bedd19d07c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47e0ed11dce4aefb5b6de6db220ba94" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "356da5c18f9b4cd8810eda1841472eb6" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcd71a5e6d2b48acbd3b75bedd19d07c" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 39773 } health_report { overall_health: HEALTHY } } }
I20260621 02:48:58.671867 27493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:58.666592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57246 (local address 127.9.166.2:39773)
0621 02:48:58.666837 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:58.666842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:58.666889 (+ 47us) server_negotiation.cc:407] Connection header received
0621 02:48:58.667107 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:58.667110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:58.667194 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:58.667314 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:48:58.667734 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.668461 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.669809 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.670198 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.670284 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:58.670717 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:48:58.671208 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:48:58.671312 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:58.671484 (+ 172us) server_negotiation.cc:299] Negotiation successful
0621 02:48:58.671550 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260621 02:48:58.680086 27431 raft_consensus.cc:1261] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6 [term 1 FOLLOWER]: Refusing update from remote peer bcd71a5e6d2b48acbd3b75bedd19d07c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:48:58.680086 27164 raft_consensus.cc:1261] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94 [term 1 FOLLOWER]: Refusing update from remote peer bcd71a5e6d2b48acbd3b75bedd19d07c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:48:58.681160 27505 consensus_queue.cc:1035] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d47e0ed11dce4aefb5b6de6db220ba94" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 36547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:48:58.681510 27502 consensus_queue.cc:1035] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [LEADER]: Connected to new peer: Peer: permanent_uuid: "356da5c18f9b4cd8810eda1841472eb6" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 41919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:48:58.692835 27508 mvcc.cc:205] Tried to move back new op lower bound from 7299113528023175168 to 7299113527875309568. Current Snapshot: MvccSnapshot[applied={T|T < 7299113528023175168}]
I20260621 02:48:58.712736 27020 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:48:58.713023 27020 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:48:58.716537 27020 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80e74bb8adc44f0b8b2ca4d07eadd22b: Sending DeleteTablet for 3 replicas of tablet 6c4cb82024d44f54b23e1a1401152bc4
I20260621 02:48:58.717614 27144 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4cb82024d44f54b23e1a1401152bc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:48:58 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:44904
I20260621 02:48:58.717923 27278 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4cb82024d44f54b23e1a1401152bc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:48:58 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:57238
I20260621 02:48:58.718017 27411 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4cb82024d44f54b23e1a1401152bc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:48:58 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:40036
I20260621 02:48:58.720439 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 27081
I20260621 02:48:58.731630 27518 tablet_replica.cc:330] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94: stopping tablet replica
I20260621 02:48:58.735370 27518 raft_consensus.cc:2229] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:48:58.736055 27518 raft_consensus.cc:2258] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:48:58.737568 27520 tablet_replica.cc:330] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6: stopping tablet replica
I20260621 02:48:58.737921 27519 tablet_replica.cc:330] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c: stopping tablet replica
I20260621 02:48:58.738046 27520 raft_consensus.cc:2229] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:48:58.738343 27519 raft_consensus.cc:2229] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:48:58.738528 27520 raft_consensus.cc:2258] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:48:58.738600 27518 ts_tablet_manager.cc:1905] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:48:58.739185 27519 raft_consensus.cc:2258] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:48:58.741329 27520 ts_tablet_manager.cc:1905] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:48:58.742347 27519 ts_tablet_manager.cc:1905] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:48:58.746860 27520 ts_tablet_manager.cc:1918] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:48:58.747051 27520 log.cc:1196] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/6c4cb82024d44f54b23e1a1401152bc4
I20260621 02:48:58.747542 27520 ts_tablet_manager.cc:1939] T 6c4cb82024d44f54b23e1a1401152bc4 P 356da5c18f9b4cd8810eda1841472eb6: Deleting consensus metadata
I20260621 02:48:58.749234 27007 catalog_manager.cc:4841] TS 356da5c18f9b4cd8810eda1841472eb6 (127.9.166.3:41919): tablet 6c4cb82024d44f54b23e1a1401152bc4 (table test-table [id=b02de2954a324fe3bb59b9409b6fc294]) successfully deleted
Jun 21 02:48:58 dist-test-slave-hvgn krb5kdc[26975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:48:58.754020 27519 ts_tablet_manager.cc:1918] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:48:58.754281 27519 log.cc:1196] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/6c4cb82024d44f54b23e1a1401152bc4
I20260621 02:48:58.754498 27518 ts_tablet_manager.cc:1918] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:48:58.754657 27518 log.cc:1196] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/6c4cb82024d44f54b23e1a1401152bc4
I20260621 02:48:58.755280 27519 ts_tablet_manager.cc:1939] T 6c4cb82024d44f54b23e1a1401152bc4 P bcd71a5e6d2b48acbd3b75bedd19d07c: Deleting consensus metadata
I20260621 02:48:58.755288 27518 ts_tablet_manager.cc:1939] T 6c4cb82024d44f54b23e1a1401152bc4 P d47e0ed11dce4aefb5b6de6db220ba94: Deleting consensus metadata
I20260621 02:48:58.757583 27006 catalog_manager.cc:4841] TS d47e0ed11dce4aefb5b6de6db220ba94 (127.9.166.1:36547): tablet 6c4cb82024d44f54b23e1a1401152bc4 (table test-table [id=b02de2954a324fe3bb59b9409b6fc294]) successfully deleted
I20260621 02:48:58.757874 27006 catalog_manager.cc:4841] TS bcd71a5e6d2b48acbd3b75bedd19d07c (127.9.166.2:39773): tablet 6c4cb82024d44f54b23e1a1401152bc4 (table test-table [id=b02de2954a324fe3bb59b9409b6fc294]) successfully deleted
I20260621 02:48:58.760749 27489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:58.722187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44908 (local address 127.9.166.1:36547)
0621 02:48:58.722413 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:58.722421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:58.728254 (+ 5833us) server_negotiation.cc:407] Connection header received
0621 02:48:58.728589 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:58.728599 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:58.728747 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:58.728843 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:48:58.731861 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.733003 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.739775 (+ 6772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.740432 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.740535 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:58.753514 (+ 12979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:48:58.753529 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:48:58.753548 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:48:58.753597 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:48:58.757809 (+ 4212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:58.758728 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:58.758735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:58.758737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:58.758868 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:58.759286 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:58.759290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:58.759292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:58.759704 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:48:58.759924 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:58.760290 (+ 366us) server_negotiation.cc:299] Negotiation successful
0621 02:48:58.760451 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:48:58.761395 27204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:48:58.870424 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 27081
I20260621 02:48:58.894416 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 27215
Jun 21 02:48:58 dist-test-slave-hvgn krb5kdc[26975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.2@KRBTEST.COM
I20260621 02:48:58.918882 27493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:58.895997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57250 (local address 127.9.166.2:39773)
0621 02:48:58.896208 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:58.896218 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:58.896241 (+ 23us) server_negotiation.cc:407] Connection header received
0621 02:48:58.896308 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:58.896314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:58.896422 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:58.896526 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:48:58.898660 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.899929 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.901863 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:58.902607 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:58.902915 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:58.908166 (+ 5251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:48:58.908188 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:48:58.908242 (+ 54us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:48:58.908417 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:48:58.913742 (+ 5325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:58.914595 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:58.914613 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:58.914619 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:58.914832 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:58.915479 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:58.915486 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:58.915490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:58.916075 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:48:58.916488 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:58.918031 (+ 1543us) server_negotiation.cc:299] Negotiation successful
0621 02:48:58.918333 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 02:48:58.919793 27338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:48:59.053574 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 27215
I20260621 02:48:59.072340 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 27348
Jun 21 02:48:59 dist-test-slave-hvgn krb5kdc[26975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.3@KRBTEST.COM
I20260621 02:48:59.086730 27491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:59.073104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40050 (local address 127.9.166.3:41919)
0621 02:48:59.073296 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:59.073301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:59.073318 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:48:59.073464 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:59.073468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:59.073543 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:59.073626 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:48:59.074835 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:59.075606 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:59.076851 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:59.077249 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:59.077337 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:59.081855 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:48:59.081869 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:48:59.081886 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:48:59.081918 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:48:59.084471 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:59.085075 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:59.085082 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:59.085084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:59.085220 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:48:59.085519 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:48:59.085522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:48:59.085524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:48:59.085860 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:48:59.086067 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:59.086299 (+ 232us) server_negotiation.cc:299] Negotiation successful
0621 02:48:59.086451 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 02:48:59.087422 27471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:48:59.176209 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 27348
I20260621 02:48:59.192893 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 26988
I20260621 02:48:59.194104 27050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:48:59.309165 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 26988
2026-06-21T02:48:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4426 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-21T02:48:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:48:59Z Disabled control of system clock
I20260621 02:48:59.384294 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:42777
--webserver_interface=127.9.166.62
--webserver_port=0
--builtin_ntp_servers=127.9.166.20:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:42777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 02:48:59.709826 27529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:48:59.710115 27529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:48:59.710279 27529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:48:59.717449 27529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:48:59.717549 27529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:48:59.717620 27529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:48:59.717731 27529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:48:59.717821 27529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:48:59.726963 27529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:42777
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:42777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:48:59.728317 27529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:48:59.730098 27529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:48:59.741750 27534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:59.741709 27537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:48:59.741786 27535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:48:59.742834 27529 server_base.cc:1034] running on GCE node
I20260621 02:48:59.743636 27529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:48:59.745057 27529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:48:59.746297 27529 hybrid_clock.cc:648] HybridClock initialized: now 1782010139746247 us; error 42 us; skew 500 ppm
I20260621 02:48:59.746665 27529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:48:59.754675 27529 webserver.cc:460] Webserver started at http://127.9.166.62:33563/ using document root <none> and password file <none>
I20260621 02:48:59.755465 27529 fs_manager.cc:359] Metadata directory not provided
I20260621 02:48:59.755601 27529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:48:59.755926 27529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:48:59.758847 27529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "44452de3421044bba9728b3f6cbcdd0a"
format_stamp: "Formatted at 2026-06-21 02:48:59 on dist-test-slave-hvgn"
server_key: "bd1c50aafa433d3c20dee62ecf026aea"
server_key_iv: "782ee5a72a63dd22ecf8789c000f8fed"
server_key_version: "encryptionkey@0"
I20260621 02:48:59.759714 27529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "44452de3421044bba9728b3f6cbcdd0a"
format_stamp: "Formatted at 2026-06-21 02:48:59 on dist-test-slave-hvgn"
server_key: "bd1c50aafa433d3c20dee62ecf026aea"
server_key_iv: "782ee5a72a63dd22ecf8789c000f8fed"
server_key_version: "encryptionkey@0"
I20260621 02:48:59.764580 27529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260621 02:48:59.767844 27543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:48:59.769192 27529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:48:59.769374 27529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "44452de3421044bba9728b3f6cbcdd0a"
format_stamp: "Formatted at 2026-06-21 02:48:59 on dist-test-slave-hvgn"
server_key: "bd1c50aafa433d3c20dee62ecf026aea"
server_key_iv: "782ee5a72a63dd22ecf8789c000f8fed"
server_key_version: "encryptionkey@0"
I20260621 02:48:59.769583 27529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:48:59.790838 27529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:48:59.791699 27529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:48:59.791996 27529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:48:59.810808 27529 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:42777
I20260621 02:48:59.810817 27594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:42777 every 8 connection(s)
I20260621 02:48:59.812211 27529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:48:59.815423 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 27529
I20260621 02:48:59.815641 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:48:59.816238 9880 external_mini_cluster.cc:1428] Setting key 97367a80d06917160af4cc04e52840c0
I20260621 02:48:59.816269 27595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:48:59.830130 27595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a: Bootstrap starting.
I20260621 02:48:59.830520 27598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:48:59.818186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32768 (local address 127.9.166.62:42777)
0621 02:48:59.819255 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0621 02:48:59.819266 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:48:59.819308 (+ 42us) server_negotiation.cc:407] Connection header received
0621 02:48:59.821058 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:48:59.821080 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:48:59.821591 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:48:59.821953 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:48:59.822529 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:59.824055 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:59.825914 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:48:59.826636 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:48:59.826776 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:48:59.828100 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:48:59.828108 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:48:59.828125 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:48:59.828676 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:48:59.828759 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:48:59.828782 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:48:59.828873 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0621 02:48:59.829139 (+ 266us) server_negotiation.cc:299] Negotiation successful
0621 02:48:59.829480 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":394,"threads_started":1}
I20260621 02:48:59.833829 27595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a: Neither blocks nor log segments found. Creating new log.
I20260621 02:48:59.835217 27595 log.cc:824] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a: Log is configured to *not* fsync() on all Append() calls
I20260621 02:48:59.838619 27595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a: No bootstrap required, opened a new log
I20260621 02:48:59.847494 27595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44452de3421044bba9728b3f6cbcdd0a" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 42777 } }
I20260621 02:48:59.847924 27595 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:48:59.848045 27595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44452de3421044bba9728b3f6cbcdd0a, State: Initialized, Role: FOLLOWER
I20260621 02:48:59.848711 27595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44452de3421044bba9728b3f6cbcdd0a" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 42777 } }
I20260621 02:48:59.849004 27595 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:48:59.849174 27595 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:48:59.849368 27595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:48:59.851856 27595 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44452de3421044bba9728b3f6cbcdd0a" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 42777 } }
I20260621 02:48:59.852407 27595 leader_election.cc:302] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44452de3421044bba9728b3f6cbcdd0a; no voters:
I20260621 02:48:59.853052 27595 leader_election.cc:288] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:48:59.853263 27600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:48:59.853966 27600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a [term 1 LEADER]: Becoming Leader. State: Replica: 44452de3421044bba9728b3f6cbcdd0a, State: Running, Role: LEADER
I20260621 02:48:59.854511 27600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44452de3421044bba9728b3f6cbcdd0a" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 42777 } }
I20260621 02:48:59.855480 27595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:48:59.857648 27601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44452de3421044bba9728b3f6cbcdd0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44452de3421044bba9728b3f6cbcdd0a" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 42777 } } }
I20260621 02:48:59.858034 27601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a [sys.catalog]: This master's current role is: LEADER
I20260621 02:48:59.859248 27602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44452de3421044bba9728b3f6cbcdd0a. Latest consensus state: current_term: 1 leader_uuid: "44452de3421044bba9728b3f6cbcdd0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44452de3421044bba9728b3f6cbcdd0a" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 42777 } } }
I20260621 02:48:59.859617 27602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a [sys.catalog]: This master's current role is: LEADER
I20260621 02:48:59.871551 27607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:48:59.879678 27607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:48:59.889477 27607 catalog_manager.cc:1347] Generated new cluster ID: e7baab86238843539ad1b11cbfa3c634
I20260621 02:48:59.889576 27607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:48:59.908810 27607 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:48:59.910307 27607 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:48:59.931550 27607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a: Generated new TSK 0
I20260621 02:48:59.932626 27607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:48:59.940924 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:42777
--builtin_ntp_servers=127.9.166.20:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 02:49:00.265753 27619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:00.266033 27619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:00.266182 27619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:00.273699 27619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:00.273798 27619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:00.273977 27619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:49:00.283020 27619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:42777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:00.284322 27619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:00.286060 27619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:00.298708 27624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:00.298750 27625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:00.298787 27627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:00.299623 27619 server_base.cc:1034] running on GCE node
I20260621 02:49:00.300225 27619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:00.301177 27619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:00.302390 27619 hybrid_clock.cc:648] HybridClock initialized: now 1782010140302331 us; error 57 us; skew 500 ppm
I20260621 02:49:00.302719 27619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:49:00.305159 27619 webserver.cc:460] Webserver started at http://127.9.166.1:36153/ using document root <none> and password file <none>
I20260621 02:49:00.305835 27619 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:00.305919 27619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:00.306238 27619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:00.309075 27619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "3937976bbf2e4e048d0f2ddfa849c49f"
format_stamp: "Formatted at 2026-06-21 02:49:00 on dist-test-slave-hvgn"
server_key: "2ef81dbc1ec604cd0e6c3e06b7dc1c13"
server_key_iv: "d7884cae043cd54fa31e7ec0412eaccd"
server_key_version: "encryptionkey@0"
I20260621 02:49:00.309865 27619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "3937976bbf2e4e048d0f2ddfa849c49f"
format_stamp: "Formatted at 2026-06-21 02:49:00 on dist-test-slave-hvgn"
server_key: "2ef81dbc1ec604cd0e6c3e06b7dc1c13"
server_key_iv: "d7884cae043cd54fa31e7ec0412eaccd"
server_key_version: "encryptionkey@0"
I20260621 02:49:00.314805 27619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 02:49:00.318437 27633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:00.319917 27619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:49:00.320107 27619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "3937976bbf2e4e048d0f2ddfa849c49f"
format_stamp: "Formatted at 2026-06-21 02:49:00 on dist-test-slave-hvgn"
server_key: "2ef81dbc1ec604cd0e6c3e06b7dc1c13"
server_key_iv: "d7884cae043cd54fa31e7ec0412eaccd"
server_key_version: "encryptionkey@0"
I20260621 02:49:00.320329 27619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:00.357278 27619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:00.358134 27619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:00.358407 27619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:00.359498 27619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:49:00.360947 27619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:49:00.361034 27619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:00.361195 27619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:49:00.361299 27619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:00.394506 27619 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:46047
I20260621 02:49:00.394567 27745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:46047 every 8 connection(s)
I20260621 02:49:00.395820 27619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:49:00.397931 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 27619
I20260621 02:49:00.398118 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:49:00.398638 9880 external_mini_cluster.cc:1428] Setting key 04d2379634ec2ee72446142c9df63639
I20260621 02:49:00.401542 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:42777
--builtin_ntp_servers=127.9.166.20:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 02:49:00.408859 27748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:00.398586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:47563 (local address 127.9.166.62:42777)
0621 02:49:00.399084 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:00.399090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:00.400580 (+ 1490us) server_negotiation.cc:407] Connection header received
0621 02:49:00.402427 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:00.402432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:00.402516 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:00.402654 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:00.403762 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:00.404586 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:00.406084 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:00.406489 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:00.406626 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:00.408109 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:00.408113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:00.408117 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:00.408134 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:00.408174 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:00.408187 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:00.408264 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:00.408537 (+ 273us) server_negotiation.cc:299] Negotiation successful
0621 02:49:00.408612 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":244,"threads_started":1}
I20260621 02:49:00.410027 27749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:00.399250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:42777 (local address 127.9.166.1:47563)
0621 02:49:00.400418 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:00.400462 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:00.402176 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:00.402875 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:00.402883 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:00.402973 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:00.403602 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:00.403621 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:00.404738 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:00.404748 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:00.405943 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:00.405955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:00.406712 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:00.406721 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:00.406958 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:00.407703 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:49:00.407706 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:00.407750 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:49:00.407869 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:00.408288 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:00.408301 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:00.408310 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 02:49:00.408493 (+ 183us) client_negotiation.cc:240] Negotiation successful
0621 02:49:00.408916 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":385,"threads_started":1}
I20260621 02:49:00.411751 27746 heartbeater.cc:347] Connected to a master server at 127.9.166.62:42777
I20260621 02:49:00.412050 27746 heartbeater.cc:464] Registering TS with master...
I20260621 02:49:00.412784 27746 heartbeater.cc:511] Master 127.9.166.62:42777 requested a full tablet report, sending...
I20260621 02:49:00.414937 27560 ts_manager.cc:194] Registered new tserver with Master: 3937976bbf2e4e048d0f2ddfa849c49f (127.9.166.1:46047)
I20260621 02:49:00.417244 27560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.166.1:47563
W20260621 02:49:00.729077 27750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:00.729341 27750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:00.729426 27750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:00.736454 27750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:00.736555 27750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:00.736660 27750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:49:00.745939 27750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:42777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:00.747231 27750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:00.749040 27750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:00.763830 27756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:00.764039 27758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:00.764102 27755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:00.764839 27750 server_base.cc:1034] running on GCE node
I20260621 02:49:00.765393 27750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:00.766348 27750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:00.767546 27750 hybrid_clock.cc:648] HybridClock initialized: now 1782010140767493 us; error 44 us; skew 500 ppm
I20260621 02:49:00.767871 27750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:49:00.770062 27750 webserver.cc:460] Webserver started at http://127.9.166.2:36973/ using document root <none> and password file <none>
I20260621 02:49:00.770754 27750 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:00.770865 27750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:00.771227 27750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:00.774459 27750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "dc00943ae569416d9e9d5e16257fe947"
format_stamp: "Formatted at 2026-06-21 02:49:00 on dist-test-slave-hvgn"
server_key: "18f61bd34a9a5b387e7dab8e37435c59"
server_key_iv: "5aba6cbd6e8ce3cbedefbfc80dc1f71f"
server_key_version: "encryptionkey@0"
I20260621 02:49:00.775245 27750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "dc00943ae569416d9e9d5e16257fe947"
format_stamp: "Formatted at 2026-06-21 02:49:00 on dist-test-slave-hvgn"
server_key: "18f61bd34a9a5b387e7dab8e37435c59"
server_key_iv: "5aba6cbd6e8ce3cbedefbfc80dc1f71f"
server_key_version: "encryptionkey@0"
I20260621 02:49:00.780424 27750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260621 02:49:00.783851 27764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:00.785208 27750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260621 02:49:00.785380 27750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "dc00943ae569416d9e9d5e16257fe947"
format_stamp: "Formatted at 2026-06-21 02:49:00 on dist-test-slave-hvgn"
server_key: "18f61bd34a9a5b387e7dab8e37435c59"
server_key_iv: "5aba6cbd6e8ce3cbedefbfc80dc1f71f"
server_key_version: "encryptionkey@0"
I20260621 02:49:00.785593 27750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:00.800909 27750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:00.801664 27750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:00.801887 27750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:00.802752 27750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:49:00.804284 27750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:49:00.804376 27750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:00.804529 27750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:49:00.804610 27750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:00.837204 27750 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:37097
I20260621 02:49:00.837225 27876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:37097 every 8 connection(s)
I20260621 02:49:00.838701 27750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:49:00.846918 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 27750
I20260621 02:49:00.847141 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:49:00.847638 9880 external_mini_cluster.cc:1428] Setting key 32dc31f960b07112545781a41d697673
I20260621 02:49:00.850687 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:42777
--builtin_ntp_servers=127.9.166.20:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 02:49:00.851473 27748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:00.841299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:35217 (local address 127.9.166.62:42777)
0621 02:49:00.841487 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:00.841491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:00.842835 (+ 1344us) server_negotiation.cc:407] Connection header received
0621 02:49:00.844682 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:00.844688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:00.844778 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:00.844887 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:00.845915 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:00.846622 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:00.848441 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:00.848922 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:00.849020 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:00.850548 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:00.850554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:00.850558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:00.850581 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:00.850622 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:00.850640 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:00.850743 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:00.851051 (+ 308us) server_negotiation.cc:299] Negotiation successful
0621 02:49:00.851141 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:49:00.852427 27879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:00.841803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:42777 (local address 127.9.166.2:35217)
0621 02:49:00.842676 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:00.842715 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:00.844407 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:00.845119 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:00.845127 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:00.845213 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:00.845751 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:00.845766 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:00.846773 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:00.846784 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:00.848284 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:00.848302 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:00.849138 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:00.849145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:00.849457 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:00.850252 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:49:00.850254 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:00.850291 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:49:00.850362 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:00.850787 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:00.850799 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:00.850808 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 02:49:00.850998 (+ 190us) client_negotiation.cc:240] Negotiation successful
0621 02:49:00.851363 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":281,"threads_started":1}
I20260621 02:49:00.853790 27877 heartbeater.cc:347] Connected to a master server at 127.9.166.62:42777
I20260621 02:49:00.854111 27877 heartbeater.cc:464] Registering TS with master...
I20260621 02:49:00.854821 27877 heartbeater.cc:511] Master 127.9.166.62:42777 requested a full tablet report, sending...
I20260621 02:49:00.856271 27560 ts_manager.cc:194] Registered new tserver with Master: dc00943ae569416d9e9d5e16257fe947 (127.9.166.2:37097)
I20260621 02:49:00.857518 27560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.166.2:35217
W20260621 02:49:01.173316 27880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:01.173581 27880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:01.173748 27880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:01.180781 27880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:01.180868 27880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:01.181054 27880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:49:01.189944 27880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:42777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:01.191246 27880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:01.192968 27880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:01.205636 27885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:01.205610 27886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:01.205843 27888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:01.206297 27880 server_base.cc:1034] running on GCE node
I20260621 02:49:01.207053 27880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:01.208004 27880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:01.209236 27880 hybrid_clock.cc:648] HybridClock initialized: now 1782010141209192 us; error 57 us; skew 500 ppm
I20260621 02:49:01.209538 27880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:49:01.211753 27880 webserver.cc:460] Webserver started at http://127.9.166.3:40461/ using document root <none> and password file <none>
I20260621 02:49:01.212420 27880 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:01.212558 27880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:01.212867 27880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:01.215695 27880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "201475973f43441f9c1d328d017d781a"
format_stamp: "Formatted at 2026-06-21 02:49:01 on dist-test-slave-hvgn"
server_key: "2431ab071f4ff59397e140608cfb13da"
server_key_iv: "ba4176029f92f3cbfbfe61700b5f13e2"
server_key_version: "encryptionkey@0"
I20260621 02:49:01.216449 27880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "201475973f43441f9c1d328d017d781a"
format_stamp: "Formatted at 2026-06-21 02:49:01 on dist-test-slave-hvgn"
server_key: "2431ab071f4ff59397e140608cfb13da"
server_key_iv: "ba4176029f92f3cbfbfe61700b5f13e2"
server_key_version: "encryptionkey@0"
I20260621 02:49:01.221238 27880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:49:01.224658 27894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:01.225901 27880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:49:01.226076 27880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "201475973f43441f9c1d328d017d781a"
format_stamp: "Formatted at 2026-06-21 02:49:01 on dist-test-slave-hvgn"
server_key: "2431ab071f4ff59397e140608cfb13da"
server_key_iv: "ba4176029f92f3cbfbfe61700b5f13e2"
server_key_version: "encryptionkey@0"
I20260621 02:49:01.226280 27880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:01.249307 27880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:01.250211 27880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:01.250507 27880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:01.251591 27880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:49:01.253108 27880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:49:01.253224 27880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:01.253360 27880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:49:01.253448 27880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:01.288002 27880 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:39903
I20260621 02:49:01.288089 28006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:39903 every 8 connection(s)
I20260621 02:49:01.289322 27880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:49:01.295378 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 27880
I20260621 02:49:01.295617 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:49:01.296200 9880 external_mini_cluster.cc:1428] Setting key 0e1b812d3565dfb9bdcb6a4aa6d139f0
I20260621 02:49:01.302089 27748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.291924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:58227 (local address 127.9.166.62:42777)
0621 02:49:01.292149 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:01.292155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:01.293537 (+ 1382us) server_negotiation.cc:407] Connection header received
0621 02:49:01.295461 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:01.295479 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:01.295576 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:01.295658 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:01.296975 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.297826 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.299335 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.299826 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.299960 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.301303 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:01.301309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:01.301314 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:01.301340 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:01.301370 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:01.301382 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:01.301475 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:01.301686 (+ 211us) server_negotiation.cc:299] Negotiation successful
0621 02:49:01.301775 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260621 02:49:01.303040 28009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.292481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:42777 (local address 127.9.166.3:58227)
0621 02:49:01.293352 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:01.293393 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:01.295100 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:01.295929 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:01.295941 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:01.296072 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:01.296807 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:01.296825 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.297994 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.297998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:01.299196 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:01.299207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.300044 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.300061 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:01.300281 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.300980 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:49:01.300982 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:01.301019 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:49:01.301119 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:01.301474 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:01.301485 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:01.301494 (+ 9us) client_negotiation.cc:769] Sending connection context
0621 02:49:01.301675 (+ 181us) client_negotiation.cc:240] Negotiation successful
0621 02:49:01.302035 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":258,"threads_started":1}
I20260621 02:49:01.304351 28007 heartbeater.cc:347] Connected to a master server at 127.9.166.62:42777
I20260621 02:49:01.304634 28007 heartbeater.cc:464] Registering TS with master...
I20260621 02:49:01.305289 28007 heartbeater.cc:511] Master 127.9.166.62:42777 requested a full tablet report, sending...
I20260621 02:49:01.306697 27560 ts_manager.cc:194] Registered new tserver with Master: 201475973f43441f9c1d328d017d781a (127.9.166.3:39903)
I20260621 02:49:01.307566 27560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.166.3:58227
I20260621 02:49:01.311007 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:49:01.321076 27748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.314980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32776 (local address 127.9.166.62:42777)
0621 02:49:01.315152 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:01.315157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:01.315667 (+ 510us) server_negotiation.cc:407] Connection header received
0621 02:49:01.315816 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:01.315822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:01.315912 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:01.316001 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:01.316568 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.317272 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.318670 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.319067 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.319161 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.320455 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:01.320460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:01.320463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:01.320480 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:01.320557 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:01.320571 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:01.320645 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:01.320768 (+ 123us) server_negotiation.cc:299] Negotiation successful
0621 02:49:01.320837 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:49:01.324817 27560 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:49:01.331938 27560 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:49:01.380522 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.344010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:37097 (local address 127.0.0.1:43278)
0621 02:49:01.370407 (+ 26397us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:01.370467 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:01.370773 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:01.372653 (+ 1880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:01.372660 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:01.372697 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:01.372980 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:01.372989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.375062 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.375068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:01.377672 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:01.377691 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.379488 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.379495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:01.379694 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.379744 (+ 50us) client_negotiation.cc:769] Sending connection context
0621 02:49:01.379896 (+ 152us) client_negotiation.cc:240] Negotiation successful
0621 02:49:01.380119 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26099,"mutex_wait_us":34,"thread_start_us":28033,"threads_started":1}
I20260621 02:49:01.381223 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.343983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50256 (local address 127.9.166.3:39903)
0621 02:49:01.372763 (+ 28780us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:01.372771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:01.372810 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:49:01.372889 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:01.372894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:01.373155 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:01.373375 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:01.373963 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.375912 (+ 1949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.378342 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.379796 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.379937 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.380220 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:01.380676 (+ 456us) server_negotiation.cc:299] Negotiation successful
0621 02:49:01.380841 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28603,"thread_start_us":28411,"threads_started":1}
I20260621 02:49:01.381105 28019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.343867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:39903 (local address 127.0.0.1:50256)
0621 02:49:01.371590 (+ 27723us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:01.371613 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:01.371778 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:01.373367 (+ 1589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:01.373371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:01.373396 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:01.373768 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:01.373781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.376061 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.376067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:01.378167 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:01.378177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.380336 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.380341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:01.380476 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.380497 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 02:49:01.380619 (+ 122us) client_negotiation.cc:240] Negotiation successful
0621 02:49:01.380744 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26049,"mutex_wait_us":1,"thread_start_us":25775,"threads_started":1}
I20260621 02:49:01.381737 28023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.344492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.9.166.2:37097)
0621 02:49:01.368817 (+ 24325us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:01.368831 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:01.371041 (+ 2210us) server_negotiation.cc:407] Connection header received
0621 02:49:01.371175 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:01.371187 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:01.372116 (+ 929us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:01.372359 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:01.373186 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.374884 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.377888 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.379270 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.379448 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.379716 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:01.380924 (+ 1208us) server_negotiation.cc:299] Negotiation successful
0621 02:49:01.381117 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24029,"thread_start_us":24325,"threads_started":1}
I20260621 02:49:01.381820 28020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.344276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:46047 (local address 127.0.0.1:55036)
0621 02:49:01.372304 (+ 28028us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:01.372323 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:01.373337 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:01.374282 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:01.374285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:01.374304 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:01.374573 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:01.374580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.376539 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.376548 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:01.378211 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:01.378225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.381049 (+ 2824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.381056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:01.381230 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.381269 (+ 39us) client_negotiation.cc:769] Sending connection context
0621 02:49:01.381355 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 02:49:01.381495 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27905,"mutex_wait_us":3,"thread_start_us":25287,"threads_started":1}
I20260621 02:49:01.382690 28024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.345004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55036 (local address 127.9.166.1:46047)
0621 02:49:01.368910 (+ 23906us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:01.368923 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:01.373211 (+ 4288us) server_negotiation.cc:407] Connection header received
0621 02:49:01.373608 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:01.373617 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:01.373883 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:01.374019 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:01.375029 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.376399 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.378417 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.380546 (+ 2129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.380690 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.380899 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:01.382185 (+ 1286us) server_negotiation.cc:299] Negotiation successful
0621 02:49:01.382354 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23675,"thread_start_us":23951,"threads_started":1}
I20260621 02:49:01.385715 27812 tablet_service.cc:1467] Processing CreateTablet for tablet 8274249a375840cf9e23688a1bcdb406 (DEFAULT_TABLE table=test-table [id=8b8148dcaa9b4e0c8df9ff0cf414ee32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:49:01.385622 27942 tablet_service.cc:1467] Processing CreateTablet for tablet 8274249a375840cf9e23688a1bcdb406 (DEFAULT_TABLE table=test-table [id=8b8148dcaa9b4e0c8df9ff0cf414ee32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:49:01.386089 27681 tablet_service.cc:1467] Processing CreateTablet for tablet 8274249a375840cf9e23688a1bcdb406 (DEFAULT_TABLE table=test-table [id=8b8148dcaa9b4e0c8df9ff0cf414ee32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:49:01.386960 27812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8274249a375840cf9e23688a1bcdb406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:01.387058 27942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8274249a375840cf9e23688a1bcdb406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:01.387272 27681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8274249a375840cf9e23688a1bcdb406. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:01.404389 28027 tablet_bootstrap.cc:492] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a: Bootstrap starting.
I20260621 02:49:01.407051 28025 tablet_bootstrap.cc:492] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947: Bootstrap starting.
I20260621 02:49:01.410331 28027 tablet_bootstrap.cc:654] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:01.411037 28026 tablet_bootstrap.cc:492] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f: Bootstrap starting.
I20260621 02:49:01.411299 28025 tablet_bootstrap.cc:654] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:01.412123 28027 log.cc:824] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:01.412487 28025 log.cc:824] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:01.415102 28026 tablet_bootstrap.cc:654] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:01.415309 28025 tablet_bootstrap.cc:492] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947: No bootstrap required, opened a new log
I20260621 02:49:01.415609 28025 ts_tablet_manager.cc:1397] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260621 02:49:01.416631 28026 log.cc:824] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:01.419600 28026 tablet_bootstrap.cc:492] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f: No bootstrap required, opened a new log
I20260621 02:49:01.419872 28026 ts_tablet_manager.cc:1397] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260621 02:49:01.420151 28027 tablet_bootstrap.cc:492] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a: No bootstrap required, opened a new log
I20260621 02:49:01.420207 27746 heartbeater.cc:503] Master 127.9.166.62:42777 was elected leader, sending a full tablet report...
I20260621 02:49:01.420406 28027 ts_tablet_manager.cc:1397] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260621 02:49:01.426121 28025 raft_consensus.cc:348] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201475973f43441f9c1d328d017d781a" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 39903 } } peers { permanent_uuid: "dc00943ae569416d9e9d5e16257fe947" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37097 } } peers { permanent_uuid: "3937976bbf2e4e048d0f2ddfa849c49f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 46047 } }
I20260621 02:49:01.426553 28025 raft_consensus.cc:374] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:01.426638 28025 raft_consensus.cc:729] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc00943ae569416d9e9d5e16257fe947, State: Initialized, Role: FOLLOWER
I20260621 02:49:01.427335 28025 consensus_queue.cc:260] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201475973f43441f9c1d328d017d781a" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 39903 } } peers { permanent_uuid: "dc00943ae569416d9e9d5e16257fe947" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37097 } } peers { permanent_uuid: "3937976bbf2e4e048d0f2ddfa849c49f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 46047 } }
I20260621 02:49:01.428548 27877 heartbeater.cc:503] Master 127.9.166.62:42777 was elected leader, sending a full tablet report...
I20260621 02:49:01.429222 28025 ts_tablet_manager.cc:1428] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260621 02:49:01.429388 28026 raft_consensus.cc:348] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201475973f43441f9c1d328d017d781a" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 39903 } } peers { permanent_uuid: "dc00943ae569416d9e9d5e16257fe947" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37097 } } peers { permanent_uuid: "3937976bbf2e4e048d0f2ddfa849c49f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 46047 } }
I20260621 02:49:01.429816 28026 raft_consensus.cc:374] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:01.429924 28026 raft_consensus.cc:729] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3937976bbf2e4e048d0f2ddfa849c49f, State: Initialized, Role: FOLLOWER
I20260621 02:49:01.430541 28026 consensus_queue.cc:260] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201475973f43441f9c1d328d017d781a" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 39903 } } peers { permanent_uuid: "dc00943ae569416d9e9d5e16257fe947" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37097 } } peers { permanent_uuid: "3937976bbf2e4e048d0f2ddfa849c49f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 46047 } }
I20260621 02:49:01.431017 28027 raft_consensus.cc:348] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201475973f43441f9c1d328d017d781a" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 39903 } } peers { permanent_uuid: "dc00943ae569416d9e9d5e16257fe947" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37097 } } peers { permanent_uuid: "3937976bbf2e4e048d0f2ddfa849c49f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 46047 } }
I20260621 02:49:01.431461 28027 raft_consensus.cc:374] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:01.431586 28027 raft_consensus.cc:729] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201475973f43441f9c1d328d017d781a, State: Initialized, Role: FOLLOWER
I20260621 02:49:01.432261 28026 ts_tablet_manager.cc:1428] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260621 02:49:01.432150 28027 consensus_queue.cc:260] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201475973f43441f9c1d328d017d781a" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 39903 } } peers { permanent_uuid: "dc00943ae569416d9e9d5e16257fe947" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37097 } } peers { permanent_uuid: "3937976bbf2e4e048d0f2ddfa849c49f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 46047 } }
I20260621 02:49:01.433207 28007 heartbeater.cc:503] Master 127.9.166.62:42777 was elected leader, sending a full tablet report...
I20260621 02:49:01.433832 28027 ts_tablet_manager.cc:1428] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260621 02:49:01.466814 28033 raft_consensus.cc:482] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:49:01.467156 28033 raft_consensus.cc:504] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201475973f43441f9c1d328d017d781a" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 39903 } } peers { permanent_uuid: "dc00943ae569416d9e9d5e16257fe947" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37097 } } peers { permanent_uuid: "3937976bbf2e4e048d0f2ddfa849c49f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 46047 } }
I20260621 02:49:01.468864 28033 leader_election.cc:288] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc00943ae569416d9e9d5e16257fe947 (127.9.166.2:37097), 3937976bbf2e4e048d0f2ddfa849c49f (127.9.166.1:46047)
I20260621 02:49:01.472761 28032 raft_consensus.cc:482] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:49:01.473062 28032 raft_consensus.cc:504] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201475973f43441f9c1d328d017d781a" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 39903 } } peers { permanent_uuid: "dc00943ae569416d9e9d5e16257fe947" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37097 } } peers { permanent_uuid: "3937976bbf2e4e048d0f2ddfa849c49f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 46047 } }
I20260621 02:49:01.475028 28032 leader_election.cc:288] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 201475973f43441f9c1d328d017d781a (127.9.166.3:39903), dc00943ae569416d9e9d5e16257fe947 (127.9.166.2:37097)
I20260621 02:49:01.477273 28009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.469200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:46047 (local address 127.9.166.3:57739)
0621 02:49:01.469503 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:01.469524 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:01.469641 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:01.470093 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:01.470100 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:01.470141 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:01.470346 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:01.470359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.471896 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.471902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:01.474100 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:01.474115 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.476311 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.476321 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:01.476525 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.476559 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 02:49:01.476671 (+ 112us) client_negotiation.cc:240] Negotiation successful
0621 02:49:01.476841 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260621 02:49:01.478062 28024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.469306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:57739 (local address 127.9.166.1:46047)
0621 02:49:01.469562 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:01.469571 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:01.469601 (+ 30us) server_negotiation.cc:407] Connection header received
0621 02:49:01.469811 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:01.469817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:01.469909 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:01.470012 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:01.470489 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.471724 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.474281 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.475550 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.475684 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.475813 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:01.477485 (+ 1672us) server_negotiation.cc:299] Negotiation successful
0621 02:49:01.477686 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:49:01.478938 27701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8274249a375840cf9e23688a1bcdb406" candidate_uuid: "201475973f43441f9c1d328d017d781a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3937976bbf2e4e048d0f2ddfa849c49f" is_pre_election: true
I20260621 02:49:01.479487 27701 raft_consensus.cc:2454] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 201475973f43441f9c1d328d017d781a in term 0.
I20260621 02:49:01.480306 27895 leader_election.cc:302] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201475973f43441f9c1d328d017d781a, 3937976bbf2e4e048d0f2ddfa849c49f; no voters:
I20260621 02:49:01.480825 28033 raft_consensus.cc:2781] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:49:01.481024 28033 raft_consensus.cc:482] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:49:01.481160 28033 raft_consensus.cc:3037] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:01.482825 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.475910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:38783 (local address 127.9.166.3:39903)
0621 02:49:01.476071 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:01.476078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:01.476700 (+ 622us) server_negotiation.cc:407] Connection header received
0621 02:49:01.477089 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:01.477094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:01.477163 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:01.477262 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:01.477906 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.478892 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.481094 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.481977 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.482107 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.482207 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:01.482471 (+ 264us) server_negotiation.cc:299] Negotiation successful
0621 02:49:01.482596 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:49:01.482857 28035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.475200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:39903 (local address 127.9.166.1:38783)
0621 02:49:01.476605 (+ 1405us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:01.476622 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:01.476924 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:01.477287 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:01.477291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:01.477315 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:01.477773 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:01.477782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.479073 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.479084 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:01.480850 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:01.480862 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.482158 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.482164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:01.482322 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.482345 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 02:49:01.482443 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 02:49:01.482562 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1278,"thread_start_us":3383,"threads_started":1}
I20260621 02:49:01.483850 27962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8274249a375840cf9e23688a1bcdb406" candidate_uuid: "3937976bbf2e4e048d0f2ddfa849c49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201475973f43441f9c1d328d017d781a" is_pre_election: true
I20260621 02:49:01.484670 28023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.469410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:42751 (local address 127.9.166.2:37097)
0621 02:49:01.469607 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:01.469613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:01.470229 (+ 616us) server_negotiation.cc:407] Connection header received
0621 02:49:01.470384 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:01.470388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:01.470476 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:01.470574 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:01.471254 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.472215 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.474782 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.476006 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.476124 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.476217 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:01.484292 (+ 8075us) server_negotiation.cc:299] Negotiation successful
0621 02:49:01.484451 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 02:49:01.484669 28034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.469230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:37097 (local address 127.9.166.3:42751)
0621 02:49:01.470122 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:01.470138 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:01.470252 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:01.470625 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:01.470630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:01.470657 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:01.471050 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:01.471062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.472376 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.472381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:01.474611 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:01.474622 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.483877 (+ 9255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.483885 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:01.484094 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.484137 (+ 43us) client_negotiation.cc:769] Sending connection context
0621 02:49:01.484230 (+ 93us) client_negotiation.cc:240] Negotiation successful
0621 02:49:01.484366 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":525,"threads_started":1}
I20260621 02:49:01.485307 28033 raft_consensus.cc:504] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201475973f43441f9c1d328d017d781a" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 39903 } } peers { permanent_uuid: "dc00943ae569416d9e9d5e16257fe947" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37097 } } peers { permanent_uuid: "3937976bbf2e4e048d0f2ddfa849c49f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 46047 } }
I20260621 02:49:01.485932 27962 raft_consensus.cc:2379] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3937976bbf2e4e048d0f2ddfa849c49f in current term 1: Already voted for candidate 201475973f43441f9c1d328d017d781a in this term.
I20260621 02:49:01.486598 28033 leader_election.cc:288] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [CANDIDATE]: Term 1 election: Requested vote from peers dc00943ae569416d9e9d5e16257fe947 (127.9.166.2:37097), 3937976bbf2e4e048d0f2ddfa849c49f (127.9.166.1:46047)
I20260621 02:49:01.487221 27701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8274249a375840cf9e23688a1bcdb406" candidate_uuid: "201475973f43441f9c1d328d017d781a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3937976bbf2e4e048d0f2ddfa849c49f"
I20260621 02:49:01.487461 27701 raft_consensus.cc:3037] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:01.489873 27831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8274249a375840cf9e23688a1bcdb406" candidate_uuid: "201475973f43441f9c1d328d017d781a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc00943ae569416d9e9d5e16257fe947"
I20260621 02:49:01.490312 27831 raft_consensus.cc:3037] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:01.490907 27701 raft_consensus.cc:2454] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201475973f43441f9c1d328d017d781a in term 1.
I20260621 02:49:01.491348 27895 leader_election.cc:302] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201475973f43441f9c1d328d017d781a, 3937976bbf2e4e048d0f2ddfa849c49f; no voters:
I20260621 02:49:01.491658 28033 raft_consensus.cc:2781] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:01.492388 28037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.476573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:37097 (local address 127.9.166.1:42171)
0621 02:49:01.478675 (+ 2102us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:01.478695 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:01.478845 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:01.485175 (+ 6330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:01.485179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:01.485203 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:01.485570 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:01.485601 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.487328 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.487332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:01.489666 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:01.489679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.491664 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.491668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:01.491840 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.491874 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 02:49:01.491974 (+ 100us) client_negotiation.cc:240] Negotiation successful
0621 02:49:01.492100 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1980,"thread_start_us":1798,"threads_started":1}
I20260621 02:49:01.492412 28023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.476124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:42171 (local address 127.9.166.2:37097)
0621 02:49:01.484903 (+ 8779us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:01.484908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:01.484924 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:49:01.484994 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:01.484998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:01.485059 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:01.485161 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:01.485760 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.486920 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.489900 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.491291 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.491370 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.491517 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:01.492005 (+ 488us) server_negotiation.cc:299] Negotiation successful
0621 02:49:01.492127 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8684,"thread_start_us":12853,"threads_started":1}
I20260621 02:49:01.494330 27831 raft_consensus.cc:2454] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 201475973f43441f9c1d328d017d781a in term 1.
I20260621 02:49:01.489830 27832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8274249a375840cf9e23688a1bcdb406" candidate_uuid: "201475973f43441f9c1d328d017d781a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc00943ae569416d9e9d5e16257fe947" is_pre_election: true
I20260621 02:49:01.495162 27832 raft_consensus.cc:2362] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 201475973f43441f9c1d328d017d781a in term 1. Re-sending same reply.
I20260621 02:49:01.495661 28033 raft_consensus.cc:686] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [term 1 LEADER]: Becoming Leader. State: Replica: 201475973f43441f9c1d328d017d781a, State: Running, Role: LEADER
I20260621 02:49:01.496251 27832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8274249a375840cf9e23688a1bcdb406" candidate_uuid: "3937976bbf2e4e048d0f2ddfa849c49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc00943ae569416d9e9d5e16257fe947" is_pre_election: true
I20260621 02:49:01.496222 28033 consensus_queue.cc:237] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201475973f43441f9c1d328d017d781a" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 39903 } } peers { permanent_uuid: "dc00943ae569416d9e9d5e16257fe947" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37097 } } peers { permanent_uuid: "3937976bbf2e4e048d0f2ddfa849c49f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 46047 } }
I20260621 02:49:01.496481 27832 raft_consensus.cc:2379] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3937976bbf2e4e048d0f2ddfa849c49f in current term 1: Already voted for candidate 201475973f43441f9c1d328d017d781a in this term.
I20260621 02:49:01.497164 27636 leader_election.cc:302] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3937976bbf2e4e048d0f2ddfa849c49f; no voters: 201475973f43441f9c1d328d017d781a, dc00943ae569416d9e9d5e16257fe947
I20260621 02:49:01.497597 28032 raft_consensus.cc:2726] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260621 02:49:01.500548 27560 catalog_manager.cc:5495] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a reported cstate change: term changed from 0 to 1, leader changed from <none> to 201475973f43441f9c1d328d017d781a (127.9.166.3). New cstate: current_term: 1 leader_uuid: "201475973f43441f9c1d328d017d781a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201475973f43441f9c1d328d017d781a" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 39903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc00943ae569416d9e9d5e16257fe947" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3937976bbf2e4e048d0f2ddfa849c49f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 46047 } health_report { overall_health: UNKNOWN } } }
W20260621 02:49:01.541054 28008 tablet.cc:2369] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:49:01.543092 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.537153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50260 (local address 127.9.166.3:39903)
0621 02:49:01.537448 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:01.537456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:01.537477 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:49:01.537957 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:01.537963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:01.538070 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:01.538231 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:49:01.538622 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.539411 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.540821 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.541391 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.541530 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.542008 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:49:01.542502 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:49:01.542673 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:01.542725 (+ 52us) server_negotiation.cc:299] Negotiation successful
0621 02:49:01.542801 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260621 02:49:01.551105 27832 raft_consensus.cc:1261] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947 [term 1 FOLLOWER]: Refusing update from remote peer 201475973f43441f9c1d328d017d781a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:49:01.551136 27701 raft_consensus.cc:1261] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f [term 1 FOLLOWER]: Refusing update from remote peer 201475973f43441f9c1d328d017d781a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:49:01.552013 28038 consensus_queue.cc:1035] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc00943ae569416d9e9d5e16257fe947" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:49:01.552354 28033 consensus_queue.cc:1035] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3937976bbf2e4e048d0f2ddfa849c49f" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:49:01.563612 28041 mvcc.cc:205] Tried to move back new op lower bound from 7299113539783680000 to 7299113539572801536. Current Snapshot: MvccSnapshot[applied={T|T < 7299113539783680000}]
I20260621 02:49:01.569068 28042 mvcc.cc:205] Tried to move back new op lower bound from 7299113539783680000 to 7299113539572801536. Current Snapshot: MvccSnapshot[applied={T|T < 7299113539783680000 or (T in {7299113539783680000})}]
I20260621 02:49:01.577111 27559 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:49:01.577353 27559 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:49:01.580858 27559 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44452de3421044bba9728b3f6cbcdd0a: Sending DeleteTablet for 3 replicas of tablet 8274249a375840cf9e23688a1bcdb406
I20260621 02:49:01.581881 27942 tablet_service.cc:1514] Processing DeleteTablet for tablet 8274249a375840cf9e23688a1bcdb406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:49:01 UTC) from {username='slave'} at 127.0.0.1:50256
I20260621 02:49:01.581933 27812 tablet_service.cc:1514] Processing DeleteTablet for tablet 8274249a375840cf9e23688a1bcdb406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:49:01 UTC) from {username='slave'} at 127.0.0.1:43278
I20260621 02:49:01.582235 27681 tablet_service.cc:1514] Processing DeleteTablet for tablet 8274249a375840cf9e23688a1bcdb406 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:49:01 UTC) from {username='slave'} at 127.0.0.1:55036
I20260621 02:49:01.584118 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 27619
I20260621 02:49:01.595471 28052 tablet_replica.cc:330] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f: stopping tablet replica
I20260621 02:49:01.595818 28051 tablet_replica.cc:330] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947: stopping tablet replica
I20260621 02:49:01.596349 28051 raft_consensus.cc:2229] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:49:01.596911 28051 raft_consensus.cc:2258] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:49:01.597605 28050 tablet_replica.cc:330] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a: stopping tablet replica
I20260621 02:49:01.597646 28052 raft_consensus.cc:2229] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:49:01.597981 28050 raft_consensus.cc:2229] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:49:01.598326 28052 raft_consensus.cc:2258] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:49:01.598618 28050 raft_consensus.cc:2258] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:49:01.599256 28051 ts_tablet_manager.cc:1905] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:49:01.600452 28052 ts_tablet_manager.cc:1905] T 8274249a375840cf9e23688a1bcdb406 P 3937976bbf2e4e048d0f2ddfa849c49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:49:01.600872 28050 ts_tablet_manager.cc:1905] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:49:01.606129 28051 ts_tablet_manager.cc:1918] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:49:01.606290 28051 log.cc:1196] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/8274249a375840cf9e23688a1bcdb406
I20260621 02:49:01.606904 28051 ts_tablet_manager.cc:1939] T 8274249a375840cf9e23688a1bcdb406 P dc00943ae569416d9e9d5e16257fe947: Deleting consensus metadata
I20260621 02:49:01.608233 28050 ts_tablet_manager.cc:1918] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:49:01.608389 28050 log.cc:1196] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/8274249a375840cf9e23688a1bcdb406
I20260621 02:49:01.608971 28050 ts_tablet_manager.cc:1939] T 8274249a375840cf9e23688a1bcdb406 P 201475973f43441f9c1d328d017d781a: Deleting consensus metadata
I20260621 02:49:01.609100 27546 catalog_manager.cc:4841] TS dc00943ae569416d9e9d5e16257fe947 (127.9.166.2:37097): tablet 8274249a375840cf9e23688a1bcdb406 (table test-table [id=8b8148dcaa9b4e0c8df9ff0cf414ee32]) successfully deleted
I20260621 02:49:01.610421 27547 catalog_manager.cc:4841] TS 201475973f43441f9c1d328d017d781a (127.9.166.3:39903): tablet 8274249a375840cf9e23688a1bcdb406 (table test-table [id=8b8148dcaa9b4e0c8df9ff0cf414ee32]) successfully deleted
I20260621 02:49:01.610743 28024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.585018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55050 (local address 127.9.166.1:46047)
0621 02:49:01.585183 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:01.585189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:01.592998 (+ 7809us) server_negotiation.cc:407] Connection header received
0621 02:49:01.593857 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:01.593865 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:01.594002 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:01.594081 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:01.595748 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.596890 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.606437 (+ 9547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.607128 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.607229 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.609022 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:01.609036 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:01.609052 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:01.609788 (+ 736us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:01.609843 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:01.609867 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:01.609976 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:01.610261 (+ 285us) server_negotiation.cc:299] Negotiation successful
0621 02:49:01.610381 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:49:01.611653 27741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:01.705603 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 27619
I20260621 02:49:01.724788 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 27750
W20260621 02:49:01.724987 27544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.166.1:46047 (error 108)
I20260621 02:49:01.727607 28020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.727100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:46047 (local address 127.0.0.1:55054)
0621 02:49:01.727284 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:01.727434 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:46047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260621 02:49:01.727952 27544 catalog_manager.cc:4584] TS 3937976bbf2e4e048d0f2ddfa849c49f (127.9.166.1:46047): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8274249a375840cf9e23688a1bcdb406: Network error: Client connection negotiation failed: client connection to 127.9.166.1:46047: connect: Connection refused (error 111)
I20260621 02:49:01.733296 28023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.725890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43294 (local address 127.9.166.2:37097)
0621 02:49:01.726084 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:01.726090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:01.726110 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:49:01.726338 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:01.726344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:01.726460 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:01.726576 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:01.727262 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.728384 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.729828 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.730305 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.730462 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.731752 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:01.731763 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:01.731781 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:01.732533 (+ 752us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:01.732594 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:01.732617 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:01.732741 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:01.732866 (+ 125us) server_negotiation.cc:299] Negotiation successful
0621 02:49:01.732983 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 02:49:01.734040 27872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:01.781876 28020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.781337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:46047 (local address 127.0.0.1:55064)
0621 02:49:01.781564 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:01.781674 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:46047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260621 02:49:01.783062 28020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.782642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:46047 (local address 127.0.0.1:55078)
0621 02:49:01.782816 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:01.782907 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:46047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260621 02:49:01.843796 28020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.843174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:46047 (local address 127.0.0.1:55090)
0621 02:49:01.843464 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:01.843588 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:46047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260621 02:49:01.845036 28020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.844606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:46047 (local address 127.0.0.1:55106)
0621 02:49:01.844783 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:01.844881 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:46047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260621 02:49:01.849403 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 27750
I20260621 02:49:01.868687 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 27880
I20260621 02:49:01.876943 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.869526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50268 (local address 127.9.166.3:39903)
0621 02:49:01.869719 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:01.869726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:01.869804 (+ 78us) server_negotiation.cc:407] Connection header received
0621 02:49:01.870079 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:01.870084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:01.870201 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:01.870324 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:01.870759 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.871761 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.873400 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:01.873896 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:01.874061 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:01.875234 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:01.875244 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:01.875260 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:01.876114 (+ 854us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:01.876184 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:01.876206 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:01.876343 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:01.876494 (+ 151us) server_negotiation.cc:299] Negotiation successful
0621 02:49:01.876604 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:49:01.877744 28002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:01.925921 28020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.925434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:46047 (local address 127.0.0.1:55118)
0621 02:49:01.925674 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:01.925775 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:46047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260621 02:49:01.926961 28020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:01.926566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:46047 (local address 127.0.0.1:55128)
0621 02:49:01.926766 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:01.926847 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:46047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260621 02:49:02.003515 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 27880
I20260621 02:49:02.019694 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 27529
I20260621 02:49:02.020802 27590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:02.131778 28020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:02.131104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:46047 (local address 127.0.0.1:55142)
0621 02:49:02.131487 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:02.131608 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:46047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260621 02:49:02.131783 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 27529
2026-06-21T02:49:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2823 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:02 dist-test-slave-hvgn krb5kdc[28061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:02 dist-test-slave-hvgn krb5kdc[28061](info): set up 2 sockets
Jun 21 02:49:02 dist-test-slave-hvgn krb5kdc[28061](info): commencing operation
krb5kdc: starting...
W20260621 02:49:03.223506 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:03 dist-test-slave-hvgn krb5kdc[28061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:49:03.466671 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:40639
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:40639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:03.793663 28074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:03.794001 28074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:03.794173 28074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:03.801328 28074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:03.801486 28074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:03.801587 28074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:03.801692 28074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:03.801779 28074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:03.811551 28074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:40639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:40639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:03.812870 28074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:03.814188 28074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:03.825803 28080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:03.825889 28082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:03.825851 28079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:03.827059 28074 server_base.cc:1034] running on GCE node
I20260621 02:49:03.827966 28074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:03.829355 28074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:03.830564 28074 hybrid_clock.cc:648] HybridClock initialized: now 1782010143830498 us; error 50 us; skew 500 ppm
I20260621 02:49:03.830932 28074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:03 dist-test-slave-hvgn krb5kdc[28061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010143, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:03.834103 28074 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:49:03.835666 28074 webserver.cc:460] Webserver started at http://127.9.166.62:36911/ using document root <none> and password file <none>
I20260621 02:49:03.836354 28074 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:03.836489 28074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:03.836797 28074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:03.839810 28074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "c3195cfa87db4d829c49b66a9e5904ec"
format_stamp: "Formatted at 2026-06-21 02:49:03 on dist-test-slave-hvgn"
server_key: "b4ed03ad20a2b89fd0e73999dcd5c230"
server_key_iv: "578440e814af956c216fd4aa4d1fdd60"
server_key_version: "encryptionkey@0"
I20260621 02:49:03.840590 28074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "c3195cfa87db4d829c49b66a9e5904ec"
format_stamp: "Formatted at 2026-06-21 02:49:03 on dist-test-slave-hvgn"
server_key: "b4ed03ad20a2b89fd0e73999dcd5c230"
server_key_iv: "578440e814af956c216fd4aa4d1fdd60"
server_key_version: "encryptionkey@0"
I20260621 02:49:03.845721 28074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:49:03.849202 28089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:03.850463 28074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:49:03.850644 28074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "c3195cfa87db4d829c49b66a9e5904ec"
format_stamp: "Formatted at 2026-06-21 02:49:03 on dist-test-slave-hvgn"
server_key: "b4ed03ad20a2b89fd0e73999dcd5c230"
server_key_iv: "578440e814af956c216fd4aa4d1fdd60"
server_key_version: "encryptionkey@0"
I20260621 02:49:03.850840 28074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:03.890547 28074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:03.895038 28074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:03.895360 28074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:03.914638 28074 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:40639
I20260621 02:49:03.914662 28140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:40639 every 8 connection(s)
I20260621 02:49:03.916109 28074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:03.920120 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 28074
I20260621 02:49:03.920186 28141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:03.920326 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:03.920926 9880 external_mini_cluster.cc:1428] Setting key 9ec729870a8892b5facd13b3f6ffe81a
I20260621 02:49:03.933158 28141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec: Bootstrap starting.
Jun 21 02:49:03 dist-test-slave-hvgn krb5kdc[28061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:49:03.936668 28141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:03.937922 28141 log.cc:824] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:03.940928 28141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec: No bootstrap required, opened a new log
I20260621 02:49:03.943753 28144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:03.922870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58508 (local address 127.9.166.62:40639)
0621 02:49:03.923802 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:03.923813 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:03.923856 (+ 43us) server_negotiation.cc:407] Connection header received
0621 02:49:03.925547 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:03.925576 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:03.925983 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:03.926417 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:03.927974 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:03.929230 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:03.930574 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:03.931293 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:03.931502 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:03.936131 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:03.936139 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:03.936154 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:03.936190 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:03.939361 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:03.940126 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:03.940141 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:03.940147 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:03.940312 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:03.940713 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:03.940719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:03.940722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:03.941297 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:03.941647 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:03.942059 (+ 412us) server_negotiation.cc:299] Negotiation successful
0621 02:49:03.942512 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":440,"threads_started":1}
I20260621 02:49:03.949744 28141 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3195cfa87db4d829c49b66a9e5904ec" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40639 } }
I20260621 02:49:03.950137 28141 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:03.950254 28141 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3195cfa87db4d829c49b66a9e5904ec, State: Initialized, Role: FOLLOWER
I20260621 02:49:03.950881 28141 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3195cfa87db4d829c49b66a9e5904ec" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40639 } }
I20260621 02:49:03.951123 28141 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:03.951277 28141 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:03.951630 28141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:03.954442 28141 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3195cfa87db4d829c49b66a9e5904ec" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40639 } }
I20260621 02:49:03.955050 28141 leader_election.cc:302] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3195cfa87db4d829c49b66a9e5904ec; no voters:
I20260621 02:49:03.955760 28141 leader_election.cc:288] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:03.955912 28146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:03.957222 28146 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec [term 1 LEADER]: Becoming Leader. State: Replica: c3195cfa87db4d829c49b66a9e5904ec, State: Running, Role: LEADER
I20260621 02:49:03.957825 28146 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3195cfa87db4d829c49b66a9e5904ec" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40639 } }
I20260621 02:49:03.958369 28141 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:03.961208 28147 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3195cfa87db4d829c49b66a9e5904ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3195cfa87db4d829c49b66a9e5904ec" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40639 } } }
I20260621 02:49:03.961665 28147 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:03.961198 28148 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3195cfa87db4d829c49b66a9e5904ec. Latest consensus state: current_term: 1 leader_uuid: "c3195cfa87db4d829c49b66a9e5904ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3195cfa87db4d829c49b66a9e5904ec" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40639 } } }
I20260621 02:49:03.961843 28148 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:03.964308 28154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:03.968695 28154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:03.977281 28154 catalog_manager.cc:1347] Generated new cluster ID: cf72d1b5dbe04c5c8d1503e5300afb96
I20260621 02:49:03.977414 28154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:03.986685 28154 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:03.988123 28154 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:03.995776 28154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec: Generated new TSK 0
I20260621 02:49:03.996656 28154 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
I20260621 02:49:04.051960 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:40639
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:04.379691 28167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:04.379976 28167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:04.380141 28167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:04.387537 28167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:04.387665 28167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:04.387845 28167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:49:04.397516 28167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:40639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:04.398738 28167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:04.400185 28167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:04.413095 28172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:04.413157 28173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:04.413228 28175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:04.414060 28167 server_base.cc:1034] running on GCE node
I20260621 02:49:04.414628 28167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:04.415584 28167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:04.416810 28167 hybrid_clock.cc:648] HybridClock initialized: now 1782010144416766 us; error 56 us; skew 500 ppm
I20260621 02:49:04.417121 28167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:04 dist-test-slave-hvgn krb5kdc[28061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010144, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:04.420310 28167 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:49:04.421813 28167 webserver.cc:460] Webserver started at http://127.9.166.1:45029/ using document root <none> and password file <none>
I20260621 02:49:04.422483 28167 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:04.422623 28167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:04.422941 28167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:04.425832 28167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "fa44db950da34f848ebf884c067b11fe"
format_stamp: "Formatted at 2026-06-21 02:49:04 on dist-test-slave-hvgn"
server_key: "2fef4cc1dfbd9cc3e7f31c35a2043ad9"
server_key_iv: "af997220f0ada6d1be5f336b39e4e819"
server_key_version: "encryptionkey@0"
I20260621 02:49:04.426615 28167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "fa44db950da34f848ebf884c067b11fe"
format_stamp: "Formatted at 2026-06-21 02:49:04 on dist-test-slave-hvgn"
server_key: "2fef4cc1dfbd9cc3e7f31c35a2043ad9"
server_key_iv: "af997220f0ada6d1be5f336b39e4e819"
server_key_version: "encryptionkey@0"
I20260621 02:49:04.431643 28167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260621 02:49:04.435451 28182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:04.436734 28167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:49:04.436902 28167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "fa44db950da34f848ebf884c067b11fe"
format_stamp: "Formatted at 2026-06-21 02:49:04 on dist-test-slave-hvgn"
server_key: "2fef4cc1dfbd9cc3e7f31c35a2043ad9"
server_key_iv: "af997220f0ada6d1be5f336b39e4e819"
server_key_version: "encryptionkey@0"
I20260621 02:49:04.437100 28167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:04.448197 28167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:04.452529 28167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:04.452801 28167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:04.453692 28167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:49:04.455173 28167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:49:04.455265 28167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:04.455421 28167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:49:04.455485 28167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260621 02:49:04.487630 28167 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:41543
I20260621 02:49:04.487668 28294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:41543 every 8 connection(s)
I20260621 02:49:04.488888 28167 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:49:04.495740 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 28167
I20260621 02:49:04.495935 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:49:04.496395 9880 external_mini_cluster.cc:1428] Setting key 05c566ebf597b6e9cdd9361f882e10f3
Jun 21 02:49:04 dist-test-slave-hvgn krb5kdc[28061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010144, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:49:04.510213 28297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:04.491436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:45399 (local address 127.9.166.62:40639)
0621 02:49:04.491856 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:04.491861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:04.492809 (+ 948us) server_negotiation.cc:407] Connection header received
0621 02:49:04.494560 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:04.494564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:04.494629 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:04.494722 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:04.496470 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:04.497498 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:04.499348 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:04.500139 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:04.500288 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:04.503617 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:04.503626 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:04.503634 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:04.503680 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:04.506995 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:04.508013 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:04.508016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:04.508019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:04.508138 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:04.508598 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:04.508601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:04.508603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:04.508841 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:04.508947 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:04.509802 (+ 855us) server_negotiation.cc:299] Negotiation successful
0621 02:49:04.509959 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":214,"threads_started":1}
I20260621 02:49:04.511732 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:04.491990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:40639 (local address 127.9.166.1:45399)
0621 02:49:04.492657 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:04.492697 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:04.494317 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:04.494968 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:04.494977 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:04.495600 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:04.496294 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:04.496313 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:04.497657 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:04.497662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:04.499200 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:04.499212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:04.500335 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:04.500340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:04.500528 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:04.501200 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:04.501222 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:04.503340 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:04.507145 (+ 3805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:04.507157 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:04.507176 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:04.507890 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:04.508242 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:04.508250 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:04.508256 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:04.508489 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:04.508957 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:04.508968 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:04.509413 (+ 445us) client_negotiation.cc:769] Sending connection context
0621 02:49:04.509708 (+ 295us) client_negotiation.cc:240] Negotiation successful
0621 02:49:04.510280 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":270,"threads_started":1}
I20260621 02:49:04.513399 28295 heartbeater.cc:347] Connected to a master server at 127.9.166.62:40639
I20260621 02:49:04.513712 28295 heartbeater.cc:464] Registering TS with master...
I20260621 02:49:04.514529 28295 heartbeater.cc:511] Master 127.9.166.62:40639 requested a full tablet report, sending...
I20260621 02:49:04.516896 28106 ts_manager.cc:194] Registered new tserver with Master: fa44db950da34f848ebf884c067b11fe (127.9.166.1:41543)
I20260621 02:49:04.519069 28106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:45399
WARNING: no policy specified for kudu/127.9.166.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
I20260621 02:49:04.552996 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:40639
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:04.876302 28301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:04.876585 28301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:04.876771 28301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:04.883848 28301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:04.883976 28301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:04.884176 28301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:49:04.893625 28301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:40639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:04.894889 28301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:04.896221 28301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:04.908999 28306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:04.908962 28309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:04.909042 28307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:04.910009 28301 server_base.cc:1034] running on GCE node
I20260621 02:49:04.910584 28301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:04.911540 28301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:04.912756 28301 hybrid_clock.cc:648] HybridClock initialized: now 1782010144912704 us; error 48 us; skew 500 ppm
I20260621 02:49:04.913069 28301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:04 dist-test-slave-hvgn krb5kdc[28061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010144, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:04.921466 28301 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:49:04.923218 28301 webserver.cc:460] Webserver started at http://127.9.166.2:36815/ using document root <none> and password file <none>
I20260621 02:49:04.924140 28301 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:04.924269 28301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:04.924593 28301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:04.928503 28301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "971b31bc498e4283a030fb98cd029445"
format_stamp: "Formatted at 2026-06-21 02:49:04 on dist-test-slave-hvgn"
server_key: "647a4100f8a10ed6d43229be1e797a61"
server_key_iv: "a52147049c728f198c1e96bebb98bed9"
server_key_version: "encryptionkey@0"
I20260621 02:49:04.929468 28301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "971b31bc498e4283a030fb98cd029445"
format_stamp: "Formatted at 2026-06-21 02:49:04 on dist-test-slave-hvgn"
server_key: "647a4100f8a10ed6d43229be1e797a61"
server_key_iv: "a52147049c728f198c1e96bebb98bed9"
server_key_version: "encryptionkey@0"
I20260621 02:49:04.935372 28301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260621 02:49:04.938688 28316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:04.940033 28301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 02:49:04.940201 28301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "971b31bc498e4283a030fb98cd029445"
format_stamp: "Formatted at 2026-06-21 02:49:04 on dist-test-slave-hvgn"
server_key: "647a4100f8a10ed6d43229be1e797a61"
server_key_iv: "a52147049c728f198c1e96bebb98bed9"
server_key_version: "encryptionkey@0"
I20260621 02:49:04.940412 28301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:04.963855 28301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:04.967892 28301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:04.968185 28301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:04.969159 28301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:49:04.970430 28301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:49:04.970547 28301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:04.970698 28301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:49:04.970772 28301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:05.004532 28301 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:43421
I20260621 02:49:05.004629 28428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:43421 every 8 connection(s)
I20260621 02:49:05.006029 28301 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:49:05.016409 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 28301
I20260621 02:49:05.016598 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:49:05.017061 9880 external_mini_cluster.cc:1428] Setting key 4e506b2ad28b24fcfe1803943453504b
Jun 21 02:49:05 dist-test-slave-hvgn krb5kdc[28061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010144, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:49:05.026937 28297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:05.008688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:37837 (local address 127.9.166.62:40639)
0621 02:49:05.008879 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:05.008883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:05.010053 (+ 1170us) server_negotiation.cc:407] Connection header received
0621 02:49:05.012041 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:05.012047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:05.012147 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:05.012261 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:05.013758 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.014475 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.016169 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.016668 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.016817 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:05.020894 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:05.020899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:05.020903 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:05.020935 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:05.023676 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:05.024630 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:05.024634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:05.024636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:05.024756 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:05.025246 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:05.025249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:05.025251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:05.025526 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:05.025634 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:05.026500 (+ 866us) server_negotiation.cc:299] Negotiation successful
0621 02:49:05.026679 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:49:05.028275 28431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:05.009188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:40639 (local address 127.9.166.2:37837)
0621 02:49:05.009882 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:05.009925 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:05.011761 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:05.012497 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:05.012505 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:05.013012 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:05.013581 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:05.013598 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.014629 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.014635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:05.016009 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:05.016029 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.016913 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.016921 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:05.017204 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:05.017883 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:05.017906 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:05.020669 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:05.023813 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:05.023825 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:05.023845 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:05.024510 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:05.024863 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:05.024871 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:05.024877 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:05.025135 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:05.025653 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:05.025664 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:05.026120 (+ 456us) client_negotiation.cc:769] Sending connection context
0621 02:49:05.026426 (+ 306us) client_negotiation.cc:240] Negotiation successful
0621 02:49:05.027025 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":278,"threads_started":1}
I20260621 02:49:05.029803 28429 heartbeater.cc:347] Connected to a master server at 127.9.166.62:40639
I20260621 02:49:05.030108 28429 heartbeater.cc:464] Registering TS with master...
I20260621 02:49:05.030784 28429 heartbeater.cc:511] Master 127.9.166.62:40639 requested a full tablet report, sending...
I20260621 02:49:05.032353 28106 ts_manager.cc:194] Registered new tserver with Master: 971b31bc498e4283a030fb98cd029445 (127.9.166.2:43421)
I20260621 02:49:05.033151 28106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:37837
WARNING: no policy specified for kudu/127.9.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
I20260621 02:49:05.070138 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:40639
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:05.394579 28434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:05.394848 28434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:05.394980 28434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:05.402123 28434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:05.402225 28434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:05.402390 28434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:49:05.412253 28434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:40639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:05.413517 28434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:05.414775 28434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:05.427848 28439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:05.427866 28442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:05.427822 28440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:05.429018 28434 server_base.cc:1034] running on GCE node
I20260621 02:49:05.429538 28434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:05.430500 28434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:05.431720 28434 hybrid_clock.cc:648] HybridClock initialized: now 1782010145431669 us; error 57 us; skew 500 ppm
I20260621 02:49:05.432042 28434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:05 dist-test-slave-hvgn krb5kdc[28061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010145, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:05.435173 28434 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:49:05.436702 28434 webserver.cc:460] Webserver started at http://127.9.166.3:38967/ using document root <none> and password file <none>
I20260621 02:49:05.437391 28434 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:05.437539 28434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:05.437855 28434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:05.440881 28434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "01af2611e365467d9a25ab3b1ab19286"
format_stamp: "Formatted at 2026-06-21 02:49:05 on dist-test-slave-hvgn"
server_key: "d2efeea116c1ed2f6e8857dd693b7bad"
server_key_iv: "76b38f261296b914a1705483aff9fe95"
server_key_version: "encryptionkey@0"
I20260621 02:49:05.441738 28434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "01af2611e365467d9a25ab3b1ab19286"
format_stamp: "Formatted at 2026-06-21 02:49:05 on dist-test-slave-hvgn"
server_key: "d2efeea116c1ed2f6e8857dd693b7bad"
server_key_iv: "76b38f261296b914a1705483aff9fe95"
server_key_version: "encryptionkey@0"
I20260621 02:49:05.447222 28434 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 02:49:05.450722 28449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:05.452266 28434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:49:05.452448 28434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "01af2611e365467d9a25ab3b1ab19286"
format_stamp: "Formatted at 2026-06-21 02:49:05 on dist-test-slave-hvgn"
server_key: "d2efeea116c1ed2f6e8857dd693b7bad"
server_key_iv: "76b38f261296b914a1705483aff9fe95"
server_key_version: "encryptionkey@0"
I20260621 02:49:05.452656 28434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:05.476722 28434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:05.480710 28434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:05.480999 28434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:05.481985 28434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:49:05.483291 28434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:49:05.483397 28434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:05.483565 28434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:49:05.483637 28434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:05.517633 28434 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:43689
I20260621 02:49:05.517681 28561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:43689 every 8 connection(s)
I20260621 02:49:05.518972 28434 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:49:05.522279 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 28434
I20260621 02:49:05.522481 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:49:05.522979 28295 heartbeater.cc:503] Master 127.9.166.62:40639 was elected leader, sending a full tablet report...
I20260621 02:49:05.523047 9880 external_mini_cluster.cc:1428] Setting key f8c5c48b3cebc70544a27df743115187
Jun 21 02:49:05 dist-test-slave-hvgn krb5kdc[28061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010145, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:49:05.539477 28297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:05.521563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:43351 (local address 127.9.166.62:40639)
0621 02:49:05.521752 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:05.521757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:05.523079 (+ 1322us) server_negotiation.cc:407] Connection header received
0621 02:49:05.525464 (+ 2385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:05.525469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:05.525574 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:05.525685 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:05.527220 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.528042 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.529776 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.530222 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.530326 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:05.533767 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:05.533772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:05.533775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:05.533805 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:05.536299 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:05.537029 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:05.537034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:05.537037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:05.537231 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:05.537741 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:05.537747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:05.537753 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:05.538111 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:05.538283 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:05.538974 (+ 691us) server_negotiation.cc:299] Negotiation successful
0621 02:49:05.539137 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:49:05.540457 28564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:05.522063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:40639 (local address 127.9.166.3:43351)
0621 02:49:05.522900 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:05.522955 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:05.525199 (+ 2244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:05.525875 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:05.525883 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:05.526489 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:05.527063 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:05.527080 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.528196 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.528200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:05.529622 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:05.529635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.530445 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.530450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:05.530646 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:05.531335 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:05.531357 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:05.533568 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:05.536460 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:05.536469 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:05.536487 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:05.536904 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:05.537350 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:05.537355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:05.537358 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:05.537632 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:05.538246 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:05.538255 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:05.538649 (+ 394us) client_negotiation.cc:769] Sending connection context
0621 02:49:05.538883 (+ 234us) client_negotiation.cc:240] Negotiation successful
0621 02:49:05.539267 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":368,"threads_started":1}
I20260621 02:49:05.541980 28562 heartbeater.cc:347] Connected to a master server at 127.9.166.62:40639
I20260621 02:49:05.542320 28562 heartbeater.cc:464] Registering TS with master...
I20260621 02:49:05.543013 28562 heartbeater.cc:511] Master 127.9.166.62:40639 requested a full tablet report, sending...
I20260621 02:49:05.544497 28106 ts_manager.cc:194] Registered new tserver with Master: 01af2611e365467d9a25ab3b1ab19286 (127.9.166.3:43689)
I20260621 02:49:05.545303 28106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:43351
I20260621 02:49:05.550514 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:49:05.568616 28297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:05.555012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58522 (local address 127.9.166.62:40639)
0621 02:49:05.555177 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:05.555181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:05.555592 (+ 411us) server_negotiation.cc:407] Connection header received
0621 02:49:05.555760 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:05.555764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:05.555826 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:05.555908 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:05.557430 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.558227 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.559745 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.560174 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.560275 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:05.563072 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:05.563077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:05.563081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:05.563113 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:05.566011 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:05.566733 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:05.566737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:05.566739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:05.566876 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:05.567240 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:05.567243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:05.567245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:05.567601 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:05.567805 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:05.568224 (+ 419us) server_negotiation.cc:299] Negotiation successful
0621 02:49:05.568376 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:49:05.572938 28106 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:49:05.579806 28106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:49:05.616523 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:05.592211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57078 (local address 127.9.166.3:43689)
0621 02:49:05.608848 (+ 16637us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:05.608854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:05.608905 (+ 51us) server_negotiation.cc:407] Connection header received
0621 02:49:05.609002 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:05.609009 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:05.609268 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:05.609471 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:05.610307 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.611863 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.614005 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.615337 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.615503 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:05.615755 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:05.615998 (+ 243us) server_negotiation.cc:299] Negotiation successful
0621 02:49:05.616160 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16488,"thread_start_us":16321,"threads_started":1}
I20260621 02:49:05.616693 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:05.591920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:43689 (local address 127.0.0.1:57078)
0621 02:49:05.593615 (+ 1695us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:05.593649 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:05.593820 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:05.609585 (+ 15765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:05.609592 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:05.609615 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:05.610061 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:05.610073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.612055 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.612060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:05.613834 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:05.613851 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.615552 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.615557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:05.615745 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:05.615802 (+ 57us) client_negotiation.cc:769] Sending connection context
0621 02:49:05.615954 (+ 152us) client_negotiation.cc:240] Negotiation successful
0621 02:49:05.616148 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1455,"mutex_wait_us":2,"thread_start_us":1323,"threads_started":1}
I20260621 02:49:05.617745 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:05.594445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34884 (local address 127.9.166.1:41543)
0621 02:49:05.607738 (+ 13293us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:05.607748 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:05.608321 (+ 573us) server_negotiation.cc:407] Connection header received
0621 02:49:05.609395 (+ 1074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:05.609413 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:05.609650 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:05.609814 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:05.610413 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.611510 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.614638 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.616120 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.616240 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:05.616460 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:05.617256 (+ 796us) server_negotiation.cc:299] Negotiation successful
0621 02:49:05.617442 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13114,"thread_start_us":12948,"threads_started":1}
I20260621 02:49:05.617801 28576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:05.594054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:41543 (local address 127.0.0.1:34884)
0621 02:49:05.608207 (+ 14153us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:05.608230 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:05.608362 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:05.609803 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:05.609809 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:05.609834 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:05.610261 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:05.610273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.611645 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.611650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:05.614457 (+ 2807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:05.614468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.616684 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.616690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:05.616920 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:05.617069 (+ 149us) client_negotiation.cc:769] Sending connection context
0621 02:49:05.617155 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 02:49:05.617331 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14023,"mutex_wait_us":291,"thread_start_us":13600,"threads_started":1}
I20260621 02:49:05.617822 28578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:05.592394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:43421 (local address 127.0.0.1:43984)
0621 02:49:05.606909 (+ 14515us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:05.606936 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:05.607094 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:05.609005 (+ 1911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:05.609013 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:05.609045 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:05.609502 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:05.609516 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.611756 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.611762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:05.614454 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:05.614468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.616628 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.616638 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:05.616923 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:05.617069 (+ 146us) client_negotiation.cc:769] Sending connection context
0621 02:49:05.617155 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 02:49:05.617331 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14327,"mutex_wait_us":277,"thread_start_us":15748,"threads_started":1}
I20260621 02:49:05.619097 28577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:05.592922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43984 (local address 127.9.166.2:43421)
0621 02:49:05.608208 (+ 15286us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:05.608217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:05.608251 (+ 34us) server_negotiation.cc:407] Connection header received
0621 02:49:05.608454 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:05.608463 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:05.608753 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:05.609707 (+ 954us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:05.609960 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.611632 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.614639 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.616117 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.616241 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:05.616426 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:05.618710 (+ 2284us) server_negotiation.cc:299] Negotiation successful
0621 02:49:05.618872 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15121,"thread_start_us":14983,"threads_started":1}
I20260621 02:49:05.621448 28497 tablet_service.cc:1467] Processing CreateTablet for tablet 8e04a23d57ae448cb6e266a12e0fc393 (DEFAULT_TABLE table=test-table [id=c2e258d268cc49bbbdfa82cd0572c0d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:49:05.621448 28230 tablet_service.cc:1467] Processing CreateTablet for tablet 8e04a23d57ae448cb6e266a12e0fc393 (DEFAULT_TABLE table=test-table [id=c2e258d268cc49bbbdfa82cd0572c0d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:49:05.622007 28364 tablet_service.cc:1467] Processing CreateTablet for tablet 8e04a23d57ae448cb6e266a12e0fc393 (DEFAULT_TABLE table=test-table [id=c2e258d268cc49bbbdfa82cd0572c0d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:49:05.622951 28230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e04a23d57ae448cb6e266a12e0fc393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:05.622962 28497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e04a23d57ae448cb6e266a12e0fc393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:05.623466 28364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e04a23d57ae448cb6e266a12e0fc393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:05.635599 28580 tablet_bootstrap.cc:492] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445: Bootstrap starting.
I20260621 02:49:05.637446 28581 tablet_bootstrap.cc:492] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe: Bootstrap starting.
I20260621 02:49:05.640034 28580 tablet_bootstrap.cc:654] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:05.641016 28581 tablet_bootstrap.cc:654] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:05.641417 28580 log.cc:824] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:05.642323 28581 log.cc:824] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:05.643505 28582 tablet_bootstrap.cc:492] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286: Bootstrap starting.
I20260621 02:49:05.647061 28582 tablet_bootstrap.cc:654] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:05.648358 28582 log.cc:824] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:05.650295 28580 tablet_bootstrap.cc:492] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445: No bootstrap required, opened a new log
I20260621 02:49:05.650292 28581 tablet_bootstrap.cc:492] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe: No bootstrap required, opened a new log
I20260621 02:49:05.650581 28581 ts_tablet_manager.cc:1397] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260621 02:49:05.650581 28580 ts_tablet_manager.cc:1397] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20260621 02:49:05.651010 28582 tablet_bootstrap.cc:492] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286: No bootstrap required, opened a new log
I20260621 02:49:05.651273 28582 ts_tablet_manager.cc:1397] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260621 02:49:05.659581 28582 raft_consensus.cc:348] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01af2611e365467d9a25ab3b1ab19286" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43689 } } peers { permanent_uuid: "fa44db950da34f848ebf884c067b11fe" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 41543 } } peers { permanent_uuid: "971b31bc498e4283a030fb98cd029445" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 43421 } }
I20260621 02:49:05.660015 28582 raft_consensus.cc:374] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:05.660109 28582 raft_consensus.cc:729] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01af2611e365467d9a25ab3b1ab19286, State: Initialized, Role: FOLLOWER
I20260621 02:49:05.660750 28582 consensus_queue.cc:260] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01af2611e365467d9a25ab3b1ab19286" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43689 } } peers { permanent_uuid: "fa44db950da34f848ebf884c067b11fe" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 41543 } } peers { permanent_uuid: "971b31bc498e4283a030fb98cd029445" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 43421 } }
I20260621 02:49:05.661473 28580 raft_consensus.cc:348] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01af2611e365467d9a25ab3b1ab19286" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43689 } } peers { permanent_uuid: "fa44db950da34f848ebf884c067b11fe" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 41543 } } peers { permanent_uuid: "971b31bc498e4283a030fb98cd029445" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 43421 } }
I20260621 02:49:05.661839 28562 heartbeater.cc:503] Master 127.9.166.62:40639 was elected leader, sending a full tablet report...
I20260621 02:49:05.661909 28580 raft_consensus.cc:374] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:05.662005 28580 raft_consensus.cc:729] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971b31bc498e4283a030fb98cd029445, State: Initialized, Role: FOLLOWER
I20260621 02:49:05.662560 28582 ts_tablet_manager.cc:1428] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260621 02:49:05.662611 28580 consensus_queue.cc:260] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01af2611e365467d9a25ab3b1ab19286" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43689 } } peers { permanent_uuid: "fa44db950da34f848ebf884c067b11fe" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 41543 } } peers { permanent_uuid: "971b31bc498e4283a030fb98cd029445" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 43421 } }
I20260621 02:49:05.661473 28581 raft_consensus.cc:348] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01af2611e365467d9a25ab3b1ab19286" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43689 } } peers { permanent_uuid: "fa44db950da34f848ebf884c067b11fe" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 41543 } } peers { permanent_uuid: "971b31bc498e4283a030fb98cd029445" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 43421 } }
I20260621 02:49:05.663537 28581 raft_consensus.cc:374] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:05.663630 28581 raft_consensus.cc:729] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa44db950da34f848ebf884c067b11fe, State: Initialized, Role: FOLLOWER
I20260621 02:49:05.663762 28429 heartbeater.cc:503] Master 127.9.166.62:40639 was elected leader, sending a full tablet report...
I20260621 02:49:05.664455 28580 ts_tablet_manager.cc:1428] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260621 02:49:05.664317 28581 consensus_queue.cc:260] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01af2611e365467d9a25ab3b1ab19286" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43689 } } peers { permanent_uuid: "fa44db950da34f848ebf884c067b11fe" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 41543 } } peers { permanent_uuid: "971b31bc498e4283a030fb98cd029445" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 43421 } }
I20260621 02:49:05.666177 28581 ts_tablet_manager.cc:1428] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260621 02:49:05.741372 28296 tablet.cc:2369] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:49:05.758327 28430 tablet.cc:2369] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:49:05.770614 28563 tablet.cc:2369] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:49:05.831267 28586 raft_consensus.cc:482] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:49:05.831555 28586 raft_consensus.cc:504] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01af2611e365467d9a25ab3b1ab19286" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43689 } } peers { permanent_uuid: "fa44db950da34f848ebf884c067b11fe" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 41543 } } peers { permanent_uuid: "971b31bc498e4283a030fb98cd029445" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 43421 } }
I20260621 02:49:05.833209 28586 leader_election.cc:288] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa44db950da34f848ebf884c067b11fe (127.9.166.1:41543), 971b31bc498e4283a030fb98cd029445 (127.9.166.2:43421)
I20260621 02:49:05.839048 28577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:05.833561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:55047 (local address 127.9.166.2:43421)
0621 02:49:05.833770 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:05.833777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:05.834053 (+ 276us) server_negotiation.cc:407] Connection header received
0621 02:49:05.834325 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:05.834331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:05.834412 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:05.834516 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:05.834947 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.835745 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.837412 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.838161 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.838303 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:05.838406 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:05.838671 (+ 265us) server_negotiation.cc:299] Negotiation successful
0621 02:49:05.838795 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:49:05.839077 28564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:05.833580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:43421 (local address 127.9.166.3:55047)
0621 02:49:05.833924 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:05.833953 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:05.834116 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:05.834544 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:05.834549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:05.834585 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:05.834797 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:05.834806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.835872 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.835886 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:05.837251 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:05.837263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.838318 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.838324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:05.838505 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:05.838529 (+ 24us) client_negotiation.cc:769] Sending connection context
0621 02:49:05.838632 (+ 103us) client_negotiation.cc:240] Negotiation successful
0621 02:49:05.838762 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"spinlock_wait_cycles":49408}
I20260621 02:49:05.839839 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:05.833561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:41423 (local address 127.9.166.1:41543)
0621 02:49:05.833770 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:05.833777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:05.834556 (+ 779us) server_negotiation.cc:407] Connection header received
0621 02:49:05.834836 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:05.834846 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:05.834936 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:05.835038 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:05.835627 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.836324 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.837954 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.838797 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.838960 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:05.839062 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:05.839471 (+ 409us) server_negotiation.cc:299] Negotiation successful
0621 02:49:05.839577 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260621 02:49:05.840193 28384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e04a23d57ae448cb6e266a12e0fc393" candidate_uuid: "01af2611e365467d9a25ab3b1ab19286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971b31bc498e4283a030fb98cd029445" is_pre_election: true
I20260621 02:49:05.840572 28589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:05.833612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:41543 (local address 127.9.166.3:41423)
0621 02:49:05.834442 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:05.834460 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:05.834669 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:05.835053 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:05.835058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:05.835081 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:05.835457 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:05.835466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.836489 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.836493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:05.837799 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:05.837810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.839013 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.839020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:05.839239 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:05.839283 (+ 44us) client_negotiation.cc:769] Sending connection context
0621 02:49:05.840027 (+ 744us) client_negotiation.cc:240] Negotiation successful
0621 02:49:05.840154 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"spinlock_wait_cycles":40704,"thread_start_us":454,"threads_started":1}
I20260621 02:49:05.840819 28384 raft_consensus.cc:2454] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01af2611e365467d9a25ab3b1ab19286 in term 0.
I20260621 02:49:05.841570 28451 leader_election.cc:302] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01af2611e365467d9a25ab3b1ab19286, 971b31bc498e4283a030fb98cd029445; no voters:
I20260621 02:49:05.841758 28250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e04a23d57ae448cb6e266a12e0fc393" candidate_uuid: "01af2611e365467d9a25ab3b1ab19286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa44db950da34f848ebf884c067b11fe" is_pre_election: true
I20260621 02:49:05.841997 28586 raft_consensus.cc:2781] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:49:05.842139 28586 raft_consensus.cc:482] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:49:05.842257 28586 raft_consensus.cc:3037] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:05.842370 28250 raft_consensus.cc:2454] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01af2611e365467d9a25ab3b1ab19286 in term 0.
I20260621 02:49:05.844696 28586 raft_consensus.cc:504] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01af2611e365467d9a25ab3b1ab19286" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43689 } } peers { permanent_uuid: "fa44db950da34f848ebf884c067b11fe" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 41543 } } peers { permanent_uuid: "971b31bc498e4283a030fb98cd029445" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 43421 } }
I20260621 02:49:05.845396 28586 leader_election.cc:288] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [CANDIDATE]: Term 1 election: Requested vote from peers fa44db950da34f848ebf884c067b11fe (127.9.166.1:41543), 971b31bc498e4283a030fb98cd029445 (127.9.166.2:43421)
I20260621 02:49:05.845902 28384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e04a23d57ae448cb6e266a12e0fc393" candidate_uuid: "01af2611e365467d9a25ab3b1ab19286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971b31bc498e4283a030fb98cd029445"
I20260621 02:49:05.845925 28250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e04a23d57ae448cb6e266a12e0fc393" candidate_uuid: "01af2611e365467d9a25ab3b1ab19286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa44db950da34f848ebf884c067b11fe"
I20260621 02:49:05.846097 28384 raft_consensus.cc:3037] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:05.846129 28250 raft_consensus.cc:3037] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:05.848416 28250 raft_consensus.cc:2454] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01af2611e365467d9a25ab3b1ab19286 in term 1.
I20260621 02:49:05.848446 28384 raft_consensus.cc:2454] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01af2611e365467d9a25ab3b1ab19286 in term 1.
I20260621 02:49:05.848829 28451 leader_election.cc:302] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01af2611e365467d9a25ab3b1ab19286, 971b31bc498e4283a030fb98cd029445; no voters:
I20260621 02:49:05.849170 28586 raft_consensus.cc:2781] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:05.849627 28586 raft_consensus.cc:686] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [term 1 LEADER]: Becoming Leader. State: Replica: 01af2611e365467d9a25ab3b1ab19286, State: Running, Role: LEADER
I20260621 02:49:05.850090 28586 consensus_queue.cc:237] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01af2611e365467d9a25ab3b1ab19286" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43689 } } peers { permanent_uuid: "fa44db950da34f848ebf884c067b11fe" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 41543 } } peers { permanent_uuid: "971b31bc498e4283a030fb98cd029445" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 43421 } }
I20260621 02:49:05.854756 28104 catalog_manager.cc:5495] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01af2611e365467d9a25ab3b1ab19286 (127.9.166.3). New cstate: current_term: 1 leader_uuid: "01af2611e365467d9a25ab3b1ab19286" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01af2611e365467d9a25ab3b1ab19286" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 43689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa44db950da34f848ebf884c067b11fe" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 41543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "971b31bc498e4283a030fb98cd029445" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 43421 } health_report { overall_health: UNKNOWN } } }
I20260621 02:49:05.899838 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:05.894422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57082 (local address 127.9.166.3:43689)
0621 02:49:05.894658 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:05.894666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:05.894696 (+ 30us) server_negotiation.cc:407] Connection header received
0621 02:49:05.894926 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:05.894935 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:05.895015 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:05.895141 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:49:05.895512 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.896250 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.897625 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.898057 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.898160 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:05.898620 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:49:05.899160 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:49:05.899295 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:05.899497 (+ 202us) server_negotiation.cc:299] Negotiation successful
0621 02:49:05.899552 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 02:49:05.908054 28384 raft_consensus.cc:1261] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445 [term 1 FOLLOWER]: Refusing update from remote peer 01af2611e365467d9a25ab3b1ab19286: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:49:05.908080 28250 raft_consensus.cc:1261] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe [term 1 FOLLOWER]: Refusing update from remote peer 01af2611e365467d9a25ab3b1ab19286: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:49:05.909024 28590 consensus_queue.cc:1035] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa44db950da34f848ebf884c067b11fe" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 41543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:49:05.909366 28586 consensus_queue.cc:1035] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "971b31bc498e4283a030fb98cd029445" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 43421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:49:05.923144 28593 mvcc.cc:205] Tried to move back new op lower bound from 7299113557629267968 to 7299113557405515776. Current Snapshot: MvccSnapshot[applied={T|T < 7299113557629267968}]
I20260621 02:49:05.923467 28594 mvcc.cc:205] Tried to move back new op lower bound from 7299113557629267968 to 7299113557405515776. Current Snapshot: MvccSnapshot[applied={T|T < 7299113557629267968}]
I20260621 02:49:05.929831 28595 mvcc.cc:205] Tried to move back new op lower bound from 7299113557629267968 to 7299113557405515776. Current Snapshot: MvccSnapshot[applied={T|T < 7299113557629267968 or (T in {7299113557629267968})}]
I20260621 02:49:05.939949 28104 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:49:05.940263 28104 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:49:05.944069 28104 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3195cfa87db4d829c49b66a9e5904ec: Sending DeleteTablet for 3 replicas of tablet 8e04a23d57ae448cb6e266a12e0fc393
I20260621 02:49:05.945217 28497 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e04a23d57ae448cb6e266a12e0fc393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:49:05 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:57078
I20260621 02:49:05.945396 28230 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e04a23d57ae448cb6e266a12e0fc393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:49:05 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:34884
I20260621 02:49:05.945967 28364 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e04a23d57ae448cb6e266a12e0fc393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:49:05 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:43984
I20260621 02:49:05.949563 28602 tablet_replica.cc:330] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286: stopping tablet replica
I20260621 02:49:05.950460 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 28167
I20260621 02:49:05.950419 28602 raft_consensus.cc:2229] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:49:05.951359 28602 raft_consensus.cc:2258] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:49:05.954123 28602 ts_tablet_manager.cc:1905] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:49:05.955482 28603 tablet_replica.cc:330] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe: stopping tablet replica
I20260621 02:49:05.960209 28604 tablet_replica.cc:330] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445: stopping tablet replica
I20260621 02:49:05.960472 28602 ts_tablet_manager.cc:1918] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:49:05.960647 28602 log.cc:1196] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/8e04a23d57ae448cb6e266a12e0fc393
I20260621 02:49:05.960785 28604 raft_consensus.cc:2229] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:49:05.961135 28602 ts_tablet_manager.cc:1939] T 8e04a23d57ae448cb6e266a12e0fc393 P 01af2611e365467d9a25ab3b1ab19286: Deleting consensus metadata
I20260621 02:49:05.961326 28604 raft_consensus.cc:2258] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:49:05.962772 28092 catalog_manager.cc:4841] TS 01af2611e365467d9a25ab3b1ab19286 (127.9.166.3:43689): tablet 8e04a23d57ae448cb6e266a12e0fc393 (table test-table [id=c2e258d268cc49bbbdfa82cd0572c0d2]) successfully deleted
I20260621 02:49:05.962940 28603 raft_consensus.cc:2229] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:49:05.963418 28603 raft_consensus.cc:2258] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:49:05.963724 28604 ts_tablet_manager.cc:1905] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:49:05.965533 28603 ts_tablet_manager.cc:1905] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:49:05.972333 28603 ts_tablet_manager.cc:1918] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:49:05.972553 28603 log.cc:1196] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/8e04a23d57ae448cb6e266a12e0fc393
I20260621 02:49:05.972980 28604 ts_tablet_manager.cc:1918] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:49:05.973176 28603 ts_tablet_manager.cc:1939] T 8e04a23d57ae448cb6e266a12e0fc393 P fa44db950da34f848ebf884c067b11fe: Deleting consensus metadata
I20260621 02:49:05.973187 28604 log.cc:1196] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/8e04a23d57ae448cb6e266a12e0fc393
I20260621 02:49:05.973798 28604 ts_tablet_manager.cc:1939] T 8e04a23d57ae448cb6e266a12e0fc393 P 971b31bc498e4283a030fb98cd029445: Deleting consensus metadata
I20260621 02:49:05.975090 28092 catalog_manager.cc:4841] TS fa44db950da34f848ebf884c067b11fe (127.9.166.1:41543): tablet 8e04a23d57ae448cb6e266a12e0fc393 (table test-table [id=c2e258d268cc49bbbdfa82cd0572c0d2]) successfully deleted
I20260621 02:49:05.975833 28091 catalog_manager.cc:4841] TS 971b31bc498e4283a030fb98cd029445 (127.9.166.2:43421): tablet 8e04a23d57ae448cb6e266a12e0fc393 (table test-table [id=c2e258d268cc49bbbdfa82cd0572c0d2]) successfully deleted
Jun 21 02:49:05 dist-test-slave-hvgn krb5kdc[28061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:49:05.990362 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:05.953044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34896 (local address 127.9.166.1:41543)
0621 02:49:05.953275 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:05.953285 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:05.975945 (+ 22660us) server_negotiation.cc:407] Connection header received
0621 02:49:05.976139 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:05.976145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:05.976227 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:05.976331 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:05.977553 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.978271 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.979602 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:05.980006 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:05.980138 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:05.984721 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:05.984733 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:05.984748 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:05.984778 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:05.987674 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:05.988484 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:05.988491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:05.988493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:05.988616 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:05.989021 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:05.989024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:05.989026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:05.989355 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:05.989571 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:05.989888 (+ 317us) server_negotiation.cc:299] Negotiation successful
0621 02:49:05.990040 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:49:05.991118 28290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:06.096361 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 28167
I20260621 02:49:06.119328 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 28301
Jun 21 02:49:06 dist-test-slave-hvgn krb5kdc[28061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.2@KRBTEST.COM
I20260621 02:49:06.136767 28577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:06.120302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43998 (local address 127.9.166.2:43421)
0621 02:49:06.120512 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:06.120519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:06.120543 (+ 24us) server_negotiation.cc:407] Connection header received
0621 02:49:06.120610 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:06.120618 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:06.120713 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:06.120842 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:06.122103 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:06.123273 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:06.124681 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:06.125343 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:06.125472 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:06.130057 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:06.130077 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:06.130096 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:06.130140 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:06.133876 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:06.134596 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:06.134605 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:06.134609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:06.134747 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:06.135085 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:06.135090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:06.135094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:06.135576 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:06.135878 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:06.136167 (+ 289us) server_negotiation.cc:299] Negotiation successful
0621 02:49:06.136384 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:49:06.137647 28424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:06.227607 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 28301
I20260621 02:49:06.245045 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 28434
Jun 21 02:49:06 dist-test-slave-hvgn krb5kdc[28061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.3@KRBTEST.COM
I20260621 02:49:06.263033 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:06.245908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57094 (local address 127.9.166.3:43689)
0621 02:49:06.246110 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:06.246118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:06.246146 (+ 28us) server_negotiation.cc:407] Connection header received
0621 02:49:06.246275 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:06.246280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:06.246378 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:06.246508 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:06.247935 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:06.249174 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:06.250490 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:06.251220 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:06.251356 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:06.256412 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:06.256429 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:06.256446 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:06.256489 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:06.260055 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:06.260880 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:06.260887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:06.260889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:06.261062 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:06.261502 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:06.261505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:06.261507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:06.261837 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:06.262089 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:06.262416 (+ 327us) server_negotiation.cc:299] Negotiation successful
0621 02:49:06.262579 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260621 02:49:06.263854 28557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:06.386917 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 28434
I20260621 02:49:06.408449 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 28074
I20260621 02:49:06.409533 28136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:06.521811 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 28074
2026-06-21T02:49:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4389 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-21T02:49:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:06Z Disabled control of system clock
I20260621 02:49:06.599267 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:42403
--webserver_interface=127.9.166.62
--webserver_port=0
--builtin_ntp_servers=127.9.166.20:39467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:42403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 02:49:06.929430 28613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:06.929768 28613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:06.929891 28613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:06.937325 28613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:06.937429 28613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:06.937467 28613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:06.937517 28613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:06.937559 28613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:06.947216 28613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:39467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:42403
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:42403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:06.948554 28613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:06.950335 28613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:06.962095 28619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:06.962208 28621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:06.962129 28618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:06.963143 28613 server_base.cc:1034] running on GCE node
I20260621 02:49:06.963991 28613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:06.965358 28613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:06.966571 28613 hybrid_clock.cc:648] HybridClock initialized: now 1782010146966529 us; error 46 us; skew 500 ppm
I20260621 02:49:06.966946 28613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:49:06.969309 28613 webserver.cc:460] Webserver started at http://127.9.166.62:36943/ using document root <none> and password file <none>
I20260621 02:49:06.970000 28613 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:06.970113 28613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:06.970450 28613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:06.973340 28613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "9d07d57560e643008e3c8a5701f7ac38"
format_stamp: "Formatted at 2026-06-21 02:49:06 on dist-test-slave-hvgn"
server_key: "ae716e34ac7268dacc72f8c1b49da21c"
server_key_iv: "d5dda64c9d00bd843b651cd26cb7abe0"
server_key_version: "encryptionkey@0"
I20260621 02:49:06.974129 28613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "9d07d57560e643008e3c8a5701f7ac38"
format_stamp: "Formatted at 2026-06-21 02:49:06 on dist-test-slave-hvgn"
server_key: "ae716e34ac7268dacc72f8c1b49da21c"
server_key_iv: "d5dda64c9d00bd843b651cd26cb7abe0"
server_key_version: "encryptionkey@0"
I20260621 02:49:06.979035 28613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 02:49:06.982422 28627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:06.984040 28613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260621 02:49:06.984257 28613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "9d07d57560e643008e3c8a5701f7ac38"
format_stamp: "Formatted at 2026-06-21 02:49:06 on dist-test-slave-hvgn"
server_key: "ae716e34ac7268dacc72f8c1b49da21c"
server_key_iv: "d5dda64c9d00bd843b651cd26cb7abe0"
server_key_version: "encryptionkey@0"
I20260621 02:49:06.984532 28613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:07.004172 28613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:07.005040 28613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:07.005326 28613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:07.023522 28613 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:42403
I20260621 02:49:07.023574 28678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:42403 every 8 connection(s)
I20260621 02:49:07.024879 28613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:07.028555 28679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:07.031679 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 28613
I20260621 02:49:07.031855 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:07.032395 9880 external_mini_cluster.cc:1428] Setting key 845b441e865842f0e658d2eb9eb78836
I20260621 02:49:07.041142 28679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38: Bootstrap starting.
I20260621 02:49:07.044484 28679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:07.045714 28679 log.cc:824] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:07.046422 28682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:07.034362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57512 (local address 127.9.166.62:42403)
0621 02:49:07.035472 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:07.035484 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:07.035528 (+ 44us) server_negotiation.cc:407] Connection header received
0621 02:49:07.037253 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:07.037276 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:07.037789 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:07.038215 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:07.038852 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:07.040286 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:07.041743 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:07.042455 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:07.042687 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:07.043900 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:07.043909 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:07.043923 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:07.044450 (+ 527us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:07.044572 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:07.044593 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:07.044739 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:07.044975 (+ 236us) server_negotiation.cc:299] Negotiation successful
0621 02:49:07.045291 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":343,"threads_started":1}
I20260621 02:49:07.048720 28679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38: No bootstrap required, opened a new log
I20260621 02:49:07.056106 28679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d07d57560e643008e3c8a5701f7ac38" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 42403 } }
I20260621 02:49:07.056476 28679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:07.056584 28679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d07d57560e643008e3c8a5701f7ac38, State: Initialized, Role: FOLLOWER
I20260621 02:49:07.057173 28679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d07d57560e643008e3c8a5701f7ac38" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 42403 } }
I20260621 02:49:07.057415 28679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:07.057549 28679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:07.057727 28679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:07.060184 28679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d07d57560e643008e3c8a5701f7ac38" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 42403 } }
I20260621 02:49:07.060662 28679 leader_election.cc:302] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d07d57560e643008e3c8a5701f7ac38; no voters:
I20260621 02:49:07.061208 28679 leader_election.cc:288] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:07.061357 28684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:07.062091 28684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38 [term 1 LEADER]: Becoming Leader. State: Replica: 9d07d57560e643008e3c8a5701f7ac38, State: Running, Role: LEADER
I20260621 02:49:07.062618 28684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d07d57560e643008e3c8a5701f7ac38" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 42403 } }
I20260621 02:49:07.063200 28679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:07.065510 28685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d07d57560e643008e3c8a5701f7ac38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d07d57560e643008e3c8a5701f7ac38" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 42403 } } }
I20260621 02:49:07.065970 28685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:07.065810 28686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d07d57560e643008e3c8a5701f7ac38. Latest consensus state: current_term: 1 leader_uuid: "9d07d57560e643008e3c8a5701f7ac38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d07d57560e643008e3c8a5701f7ac38" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 42403 } } }
I20260621 02:49:07.066084 28686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:07.068323 28693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:07.073031 28693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:07.080861 28693 catalog_manager.cc:1347] Generated new cluster ID: 248faa8a500944da8992f5444231b838
I20260621 02:49:07.080977 28693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:07.097985 28693 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:07.099900 28693 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:07.108536 28693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38: Generated new TSK 0
I20260621 02:49:07.109587 28693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:07.114845 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:42403
--builtin_ntp_servers=127.9.166.20:39467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 02:49:07.442328 28703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:07.442651 28703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:07.442821 28703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:07.449977 28703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:07.450114 28703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:07.450294 28703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:49:07.459184 28703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:39467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:42403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:07.460502 28703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:07.462229 28703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:07.475317 28708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:07.475410 28709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:07.475422 28711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:07.480675 28703 server_base.cc:1034] running on GCE node
I20260621 02:49:07.481261 28703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:07.482142 28703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:07.483331 28703 hybrid_clock.cc:648] HybridClock initialized: now 1782010147483276 us; error 48 us; skew 500 ppm
I20260621 02:49:07.483690 28703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:49:07.485800 28703 webserver.cc:460] Webserver started at http://127.9.166.1:36757/ using document root <none> and password file <none>
I20260621 02:49:07.486478 28703 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:07.486593 28703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:07.486927 28703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:07.490011 28703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "ea5efec512bd499e8c1640b8673fef62"
format_stamp: "Formatted at 2026-06-21 02:49:07 on dist-test-slave-hvgn"
server_key: "fd8c036232c920e76c3df877b514b58a"
server_key_iv: "3ea2c7a8a3611d72602e455cd401974b"
server_key_version: "encryptionkey@0"
I20260621 02:49:07.490780 28703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "ea5efec512bd499e8c1640b8673fef62"
format_stamp: "Formatted at 2026-06-21 02:49:07 on dist-test-slave-hvgn"
server_key: "fd8c036232c920e76c3df877b514b58a"
server_key_iv: "3ea2c7a8a3611d72602e455cd401974b"
server_key_version: "encryptionkey@0"
I20260621 02:49:07.495779 28703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260621 02:49:07.498939 28717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:07.500263 28703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:49:07.500419 28703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "ea5efec512bd499e8c1640b8673fef62"
format_stamp: "Formatted at 2026-06-21 02:49:07 on dist-test-slave-hvgn"
server_key: "fd8c036232c920e76c3df877b514b58a"
server_key_iv: "3ea2c7a8a3611d72602e455cd401974b"
server_key_version: "encryptionkey@0"
I20260621 02:49:07.500626 28703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:07.525641 28703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:07.526455 28703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:07.526736 28703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:07.527720 28703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:49:07.529126 28703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:49:07.529239 28703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:07.529364 28703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:49:07.529464 28703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:07.562999 28703 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:33623
I20260621 02:49:07.563102 28829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:33623 every 8 connection(s)
I20260621 02:49:07.564467 28703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:49:07.571244 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 28703
I20260621 02:49:07.571444 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:49:07.571931 9880 external_mini_cluster.cc:1428] Setting key d7a6294818e30acd4617d25d9f3e9fa0
I20260621 02:49:07.574848 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:42403
--builtin_ntp_servers=127.9.166.20:39467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 02:49:07.578187 28832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:07.567129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:41151 (local address 127.9.166.62:42403)
0621 02:49:07.567586 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:07.567592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:07.568887 (+ 1295us) server_negotiation.cc:407] Connection header received
0621 02:49:07.570713 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:07.570719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:07.570811 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:07.570948 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:07.572162 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:07.573087 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:07.575457 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:07.575844 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:07.575941 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:07.577502 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:07.577506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:07.577510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:07.577526 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:07.577565 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:07.577576 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:07.577642 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:07.577886 (+ 244us) server_negotiation.cc:299] Negotiation successful
0621 02:49:07.577955 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":204,"threads_started":1}
I20260621 02:49:07.579624 28833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:07.567855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:42403 (local address 127.9.166.1:41151)
0621 02:49:07.568726 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:07.568767 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:07.570432 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:07.571171 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:07.571179 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:07.571271 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:07.571961 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:07.572007 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:07.573236 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:07.573240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:07.575268 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:07.575286 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:07.576080 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:07.576089 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:07.576430 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:07.577157 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:49:07.577163 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:07.577214 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:49:07.577319 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:07.577671 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:07.577684 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:07.577695 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 02:49:07.577870 (+ 175us) client_negotiation.cc:240] Negotiation successful
0621 02:49:07.578269 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":255,"threads_started":1}
I20260621 02:49:07.581056 28830 heartbeater.cc:347] Connected to a master server at 127.9.166.62:42403
I20260621 02:49:07.581341 28830 heartbeater.cc:464] Registering TS with master...
I20260621 02:49:07.582039 28830 heartbeater.cc:511] Master 127.9.166.62:42403 requested a full tablet report, sending...
I20260621 02:49:07.584244 28644 ts_manager.cc:194] Registered new tserver with Master: ea5efec512bd499e8c1640b8673fef62 (127.9.166.1:33623)
I20260621 02:49:07.585655 28644 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.166.1:41151
W20260621 02:49:07.902804 28834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:07.903080 28834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:07.903218 28834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:07.910492 28834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:07.910584 28834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:07.910776 28834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:49:07.919564 28834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:39467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:42403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:07.920818 28834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:07.922639 28834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:07.935541 28840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:07.935673 28842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:07.935678 28839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:07.936650 28834 server_base.cc:1034] running on GCE node
I20260621 02:49:07.937234 28834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:07.938151 28834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:07.939375 28834 hybrid_clock.cc:648] HybridClock initialized: now 1782010147939339 us; error 60 us; skew 500 ppm
I20260621 02:49:07.939738 28834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:49:07.941977 28834 webserver.cc:460] Webserver started at http://127.9.166.2:45955/ using document root <none> and password file <none>
I20260621 02:49:07.942647 28834 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:07.942780 28834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:07.943104 28834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:07.946195 28834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "25b5cd734c6942f6837716094e09f3d1"
format_stamp: "Formatted at 2026-06-21 02:49:07 on dist-test-slave-hvgn"
server_key: "f161e57c03a2babdb0fd18857b111c20"
server_key_iv: "a586f0290aa092529da8c9e46e97c294"
server_key_version: "encryptionkey@0"
I20260621 02:49:07.947028 28834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "25b5cd734c6942f6837716094e09f3d1"
format_stamp: "Formatted at 2026-06-21 02:49:07 on dist-test-slave-hvgn"
server_key: "f161e57c03a2babdb0fd18857b111c20"
server_key_iv: "a586f0290aa092529da8c9e46e97c294"
server_key_version: "encryptionkey@0"
I20260621 02:49:07.952170 28834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:49:07.955530 28848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:07.956754 28834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:49:07.956936 28834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "25b5cd734c6942f6837716094e09f3d1"
format_stamp: "Formatted at 2026-06-21 02:49:07 on dist-test-slave-hvgn"
server_key: "f161e57c03a2babdb0fd18857b111c20"
server_key_iv: "a586f0290aa092529da8c9e46e97c294"
server_key_version: "encryptionkey@0"
I20260621 02:49:07.957149 28834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:07.990727 28834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:07.991629 28834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:07.991915 28834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:07.992933 28834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:49:07.994414 28834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:49:07.994535 28834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:07.994668 28834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:49:07.994757 28834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:08.028200 28834 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:35121
I20260621 02:49:08.028258 28960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:35121 every 8 connection(s)
I20260621 02:49:08.029515 28834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:49:08.032120 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 28834
I20260621 02:49:08.032276 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:49:08.032816 9880 external_mini_cluster.cc:1428] Setting key db4bcf56298890979ad732af513b360a
I20260621 02:49:08.035955 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:42403
--builtin_ntp_servers=127.9.166.20:39467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 02:49:08.044899 28832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.032270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:60457 (local address 127.9.166.62:42403)
0621 02:49:08.032476 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:08.032485 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:08.034033 (+ 1548us) server_negotiation.cc:407] Connection header received
0621 02:49:08.036524 (+ 2491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:08.036530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:08.036612 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:08.036730 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:08.038152 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.039317 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.041837 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.042227 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.042328 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:08.043992 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:08.043996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:08.044000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:08.044017 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:08.044048 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:08.044060 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:08.044131 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:08.044416 (+ 285us) server_negotiation.cc:299] Negotiation successful
0621 02:49:08.044543 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 02:49:08.045775 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.032810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:42403 (local address 127.9.166.2:60457)
0621 02:49:08.033881 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:08.033926 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:08.036208 (+ 2282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:08.036998 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:08.037012 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:08.037141 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:08.037968 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:08.037995 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.039509 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.039517 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:08.041685 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:08.041703 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.042460 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.042469 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:08.042801 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:08.043640 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:49:08.043646 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:08.043702 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:49:08.043809 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:08.044157 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:08.044170 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:08.044181 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 02:49:08.044369 (+ 188us) client_negotiation.cc:240] Negotiation successful
0621 02:49:08.044668 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":348,"threads_started":1}
I20260621 02:49:08.047288 28961 heartbeater.cc:347] Connected to a master server at 127.9.166.62:42403
I20260621 02:49:08.047621 28961 heartbeater.cc:464] Registering TS with master...
I20260621 02:49:08.048339 28961 heartbeater.cc:511] Master 127.9.166.62:42403 requested a full tablet report, sending...
I20260621 02:49:08.049767 28644 ts_manager.cc:194] Registered new tserver with Master: 25b5cd734c6942f6837716094e09f3d1 (127.9.166.2:35121)
I20260621 02:49:08.051067 28644 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.166.2:60457
W20260621 02:49:08.365145 28964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:08.365414 28964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:08.365499 28964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:08.372903 28964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:08.373008 28964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:08.373114 28964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:49:08.382123 28964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:39467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:42403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:08.383342 28964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:08.385111 28964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:08.398377 28969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:08.398397 28970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:08.398597 28972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:08.399614 28964 server_base.cc:1034] running on GCE node
I20260621 02:49:08.400221 28964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:08.401235 28964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:08.402479 28964 hybrid_clock.cc:648] HybridClock initialized: now 1782010148402446 us; error 58 us; skew 500 ppm
I20260621 02:49:08.402838 28964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:49:08.405229 28964 webserver.cc:460] Webserver started at http://127.9.166.3:33779/ using document root <none> and password file <none>
I20260621 02:49:08.405944 28964 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:08.406075 28964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:08.406402 28964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:08.409250 28964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "9535d52f2ad6471392f6400349736ea7"
format_stamp: "Formatted at 2026-06-21 02:49:08 on dist-test-slave-hvgn"
server_key: "bb149a85a5121f19958a24d738488bd6"
server_key_iv: "9e2383b9550956a156331d461302c0b3"
server_key_version: "encryptionkey@0"
I20260621 02:49:08.410033 28964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "9535d52f2ad6471392f6400349736ea7"
format_stamp: "Formatted at 2026-06-21 02:49:08 on dist-test-slave-hvgn"
server_key: "bb149a85a5121f19958a24d738488bd6"
server_key_iv: "9e2383b9550956a156331d461302c0b3"
server_key_version: "encryptionkey@0"
I20260621 02:49:08.414959 28964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 02:49:08.418309 28978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:08.419615 28964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:49:08.419785 28964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "9535d52f2ad6471392f6400349736ea7"
format_stamp: "Formatted at 2026-06-21 02:49:08 on dist-test-slave-hvgn"
server_key: "bb149a85a5121f19958a24d738488bd6"
server_key_iv: "9e2383b9550956a156331d461302c0b3"
server_key_version: "encryptionkey@0"
I20260621 02:49:08.420025 28964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:08.442780 28964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:08.443620 28964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:08.443912 28964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:08.444886 28964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:49:08.446358 28964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:49:08.446488 28964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:08.446619 28964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:49:08.446707 28964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:08.479815 28964 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:36289
I20260621 02:49:08.479898 29090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:36289 every 8 connection(s)
I20260621 02:49:08.481127 28964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:49:08.482542 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 28964
I20260621 02:49:08.482718 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:49:08.483265 9880 external_mini_cluster.cc:1428] Setting key 913eb0af8f383533bfa00efd1262a1fc
I20260621 02:49:08.493460 28832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.483819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:39409 (local address 127.9.166.62:42403)
0621 02:49:08.484026 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:08.484031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:08.485803 (+ 1772us) server_negotiation.cc:407] Connection header received
0621 02:49:08.487668 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:08.487671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:08.487734 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:08.487813 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:08.488827 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.489623 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.491103 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.491563 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.491652 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:08.492840 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:08.492845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:08.492848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:08.492865 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:08.492895 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:08.492907 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:08.492973 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:08.493157 (+ 184us) server_negotiation.cc:299] Negotiation successful
0621 02:49:08.493225 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260621 02:49:08.494413 29093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.484492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:42403 (local address 127.9.166.3:39409)
0621 02:49:08.485641 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:08.485694 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:08.487432 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:08.488012 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:08.488019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:08.488104 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:08.488639 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:08.488654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.489746 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.489750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:08.490975 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:08.490986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.491741 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.491746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:08.491946 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:08.492563 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:49:08.492566 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:08.492601 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:49:08.492671 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:08.492985 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:08.492992 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:08.492997 (+ 5us) client_negotiation.cc:769] Sending connection context
0621 02:49:08.493122 (+ 125us) client_negotiation.cc:240] Negotiation successful
0621 02:49:08.493370 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":357,"threads_started":1}
I20260621 02:49:08.495810 29091 heartbeater.cc:347] Connected to a master server at 127.9.166.62:42403
I20260621 02:49:08.496102 29091 heartbeater.cc:464] Registering TS with master...
I20260621 02:49:08.496853 29091 heartbeater.cc:511] Master 127.9.166.62:42403 requested a full tablet report, sending...
I20260621 02:49:08.498315 28643 ts_manager.cc:194] Registered new tserver with Master: 9535d52f2ad6471392f6400349736ea7 (127.9.166.3:36289)
I20260621 02:49:08.499121 28643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.166.3:39409
I20260621 02:49:08.510628 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:49:08.521731 28832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.514922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57518 (local address 127.9.166.62:42403)
0621 02:49:08.515149 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:08.515159 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:08.515680 (+ 521us) server_negotiation.cc:407] Connection header received
0621 02:49:08.515934 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:08.515940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:08.516020 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:08.516132 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:08.516673 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.517444 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.518930 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.519455 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.519617 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:08.520910 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:08.520919 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:08.520930 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:08.520950 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:08.520985 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:08.520998 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:08.521164 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:08.521373 (+ 209us) server_negotiation.cc:299] Negotiation successful
0621 02:49:08.521442 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260621 02:49:08.525858 28643 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:49:08.532667 28643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:49:08.555588 29103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.544413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:35121 (local address 127.0.0.1:52978)
0621 02:49:08.545659 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:08.545702 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:08.545887 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:08.547541 (+ 1654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:08.547549 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:08.547580 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:08.548041 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:08.548053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.550157 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.550162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:08.552664 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:08.552679 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.554578 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.554585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:08.554816 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:08.554880 (+ 64us) client_negotiation.cc:769] Sending connection context
0621 02:49:08.554982 (+ 102us) client_negotiation.cc:240] Negotiation successful
0621 02:49:08.555178 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"mutex_wait_us":1,"thread_start_us":782,"threads_started":1}
I20260621 02:49:08.557953 29105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.544596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.9.166.2:35121)
0621 02:49:08.546880 (+ 2284us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:08.546889 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:08.546920 (+ 31us) server_negotiation.cc:407] Connection header received
0621 02:49:08.547026 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:08.547035 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:08.547278 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:08.548291 (+ 1013us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:08.548528 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.550013 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.552920 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.554351 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.556492 (+ 2141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:08.556698 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:08.556781 (+ 83us) server_negotiation.cc:299] Negotiation successful
0621 02:49:08.556953 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2101,"thread_start_us":1988,"threads_started":1}
I20260621 02:49:08.558444 29106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.545093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.9.166.1:33623)
0621 02:49:08.546454 (+ 1361us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:08.546465 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:08.547174 (+ 709us) server_negotiation.cc:407] Connection header received
0621 02:49:08.547272 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:08.547279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:08.547541 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:08.547727 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:08.548638 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.551237 (+ 2599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.554804 (+ 3567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.556300 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.556457 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:08.556698 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:08.557889 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0621 02:49:08.558100 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1188,"thread_start_us":1084,"threads_started":1}
I20260621 02:49:08.558557 29104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.544511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:33623 (local address 127.0.0.1:48792)
0621 02:49:08.546934 (+ 2423us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:08.546952 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:08.547077 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:08.548012 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:08.548022 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:08.548044 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:08.548449 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:08.548462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.551430 (+ 2968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.551436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:08.554577 (+ 3141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:08.554588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.557156 (+ 2568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.557165 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:08.557354 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:08.557386 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 02:49:08.557471 (+ 85us) client_negotiation.cc:240] Negotiation successful
0621 02:49:08.557600 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2280,"thread_start_us":2304,"threads_started":1}
I20260621 02:49:08.562342 28896 tablet_service.cc:1467] Processing CreateTablet for tablet 067a0e8b35374d08b43bff957747ffdf (DEFAULT_TABLE table=test-table [id=14c464fc86a04ce6bbaecb5e34d587d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:49:08.563412 28765 tablet_service.cc:1467] Processing CreateTablet for tablet 067a0e8b35374d08b43bff957747ffdf (DEFAULT_TABLE table=test-table [id=14c464fc86a04ce6bbaecb5e34d587d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:49:08.564049 28896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 067a0e8b35374d08b43bff957747ffdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:08.564738 28765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 067a0e8b35374d08b43bff957747ffdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:08.567068 29103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.547167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:36289 (local address 127.0.0.1:35162)
0621 02:49:08.555842 (+ 8675us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:08.555861 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:08.555959 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:08.560323 (+ 4364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:08.560331 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:08.560359 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:08.560533 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:08.560546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.562475 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.562483 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:08.564318 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:08.564346 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.566231 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.566237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:08.566431 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:08.566466 (+ 35us) client_negotiation.cc:769] Sending connection context
0621 02:49:08.566538 (+ 72us) client_negotiation.cc:240] Negotiation successful
0621 02:49:08.566683 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8574,"thread_start_us":12110,"threads_started":1}
I20260621 02:49:08.569718 29108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.547593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35162 (local address 127.9.166.3:36289)
0621 02:49:08.559769 (+ 12176us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:08.559775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:08.559807 (+ 32us) server_negotiation.cc:407] Connection header received
0621 02:49:08.559887 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:08.559892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:08.560145 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:08.560398 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:08.560682 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.562289 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.564538 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.566043 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.567569 (+ 1526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:08.569129 (+ 1560us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:08.569199 (+ 70us) server_negotiation.cc:299] Negotiation successful
0621 02:49:08.569365 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11541,"thread_start_us":11337,"threads_started":1}
I20260621 02:49:08.573549 29026 tablet_service.cc:1467] Processing CreateTablet for tablet 067a0e8b35374d08b43bff957747ffdf (DEFAULT_TABLE table=test-table [id=14c464fc86a04ce6bbaecb5e34d587d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:49:08.576097 29026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 067a0e8b35374d08b43bff957747ffdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:08.578191 29109 tablet_bootstrap.cc:492] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1: Bootstrap starting.
I20260621 02:49:08.580439 29110 tablet_bootstrap.cc:492] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62: Bootstrap starting.
I20260621 02:49:08.581808 29109 tablet_bootstrap.cc:654] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:08.583055 29109 log.cc:824] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:08.583966 29110 tablet_bootstrap.cc:654] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:08.585155 29110 log.cc:824] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:08.586800 29109 tablet_bootstrap.cc:492] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1: No bootstrap required, opened a new log
I20260621 02:49:08.587055 29109 ts_tablet_manager.cc:1397] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260621 02:49:08.587675 29110 tablet_bootstrap.cc:492] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62: No bootstrap required, opened a new log
I20260621 02:49:08.587919 29110 ts_tablet_manager.cc:1397] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260621 02:49:08.588465 28830 heartbeater.cc:503] Master 127.9.166.62:42403 was elected leader, sending a full tablet report...
I20260621 02:49:08.597364 29110 raft_consensus.cc:348] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25b5cd734c6942f6837716094e09f3d1" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 35121 } } peers { permanent_uuid: "ea5efec512bd499e8c1640b8673fef62" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 33623 } } peers { permanent_uuid: "9535d52f2ad6471392f6400349736ea7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 36289 } }
I20260621 02:49:08.598119 29110 raft_consensus.cc:374] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:08.598244 29110 raft_consensus.cc:729] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5efec512bd499e8c1640b8673fef62, State: Initialized, Role: FOLLOWER
I20260621 02:49:08.598672 29113 tablet_bootstrap.cc:492] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7: Bootstrap starting.
I20260621 02:49:08.598989 29110 consensus_queue.cc:260] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25b5cd734c6942f6837716094e09f3d1" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 35121 } } peers { permanent_uuid: "ea5efec512bd499e8c1640b8673fef62" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 33623 } } peers { permanent_uuid: "9535d52f2ad6471392f6400349736ea7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 36289 } }
I20260621 02:49:08.600291 29109 raft_consensus.cc:348] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25b5cd734c6942f6837716094e09f3d1" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 35121 } } peers { permanent_uuid: "ea5efec512bd499e8c1640b8673fef62" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 33623 } } peers { permanent_uuid: "9535d52f2ad6471392f6400349736ea7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 36289 } }
I20260621 02:49:08.600739 29109 raft_consensus.cc:374] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:08.600840 29110 ts_tablet_manager.cc:1428] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
I20260621 02:49:08.600853 29109 raft_consensus.cc:729] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25b5cd734c6942f6837716094e09f3d1, State: Initialized, Role: FOLLOWER
I20260621 02:49:08.601432 29109 consensus_queue.cc:260] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25b5cd734c6942f6837716094e09f3d1" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 35121 } } peers { permanent_uuid: "ea5efec512bd499e8c1640b8673fef62" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 33623 } } peers { permanent_uuid: "9535d52f2ad6471392f6400349736ea7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 36289 } }
I20260621 02:49:08.602595 28961 heartbeater.cc:503] Master 127.9.166.62:42403 was elected leader, sending a full tablet report...
I20260621 02:49:08.603291 29109 ts_tablet_manager.cc:1428] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260621 02:49:08.603938 29113 tablet_bootstrap.cc:654] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:08.605068 29113 log.cc:824] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:08.608096 29113 tablet_bootstrap.cc:492] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7: No bootstrap required, opened a new log
I20260621 02:49:08.608369 29113 ts_tablet_manager.cc:1397] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260621 02:49:08.615298 29114 raft_consensus.cc:482] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:49:08.615110 29113 raft_consensus.cc:348] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25b5cd734c6942f6837716094e09f3d1" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 35121 } } peers { permanent_uuid: "ea5efec512bd499e8c1640b8673fef62" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 33623 } } peers { permanent_uuid: "9535d52f2ad6471392f6400349736ea7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 36289 } }
I20260621 02:49:08.615522 29113 raft_consensus.cc:374] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:08.615624 29113 raft_consensus.cc:729] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9535d52f2ad6471392f6400349736ea7, State: Initialized, Role: FOLLOWER
I20260621 02:49:08.615594 29114 raft_consensus.cc:504] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25b5cd734c6942f6837716094e09f3d1" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 35121 } } peers { permanent_uuid: "ea5efec512bd499e8c1640b8673fef62" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 33623 } } peers { permanent_uuid: "9535d52f2ad6471392f6400349736ea7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 36289 } }
I20260621 02:49:08.616199 29113 consensus_queue.cc:260] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25b5cd734c6942f6837716094e09f3d1" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 35121 } } peers { permanent_uuid: "ea5efec512bd499e8c1640b8673fef62" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 33623 } } peers { permanent_uuid: "9535d52f2ad6471392f6400349736ea7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 36289 } }
I20260621 02:49:08.617241 29091 heartbeater.cc:503] Master 127.9.166.62:42403 was elected leader, sending a full tablet report...
I20260621 02:49:08.617956 29113 ts_tablet_manager.cc:1428] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260621 02:49:08.618217 29114 leader_election.cc:288] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25b5cd734c6942f6837716094e09f3d1 (127.9.166.2:35121), 9535d52f2ad6471392f6400349736ea7 (127.9.166.3:36289)
I20260621 02:49:08.625938 29108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.618083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:35757 (local address 127.9.166.3:36289)
0621 02:49:08.618321 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:08.618330 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:08.618785 (+ 455us) server_negotiation.cc:407] Connection header received
0621 02:49:08.619101 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:08.619107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:08.619174 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:08.619258 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:08.619922 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.621055 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.624140 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.625041 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.625173 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:08.625302 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:08.625587 (+ 285us) server_negotiation.cc:299] Negotiation successful
0621 02:49:08.625702 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260621 02:49:08.625993 29105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.618104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:49213 (local address 127.9.166.2:35121)
0621 02:49:08.618493 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:08.618499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:08.619128 (+ 629us) server_negotiation.cc:407] Connection header received
0621 02:49:08.619289 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:08.619294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:08.619420 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:08.619524 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:08.620179 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.621209 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.624113 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.625112 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.625248 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:08.625344 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:08.625713 (+ 369us) server_negotiation.cc:299] Negotiation successful
0621 02:49:08.625816 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260621 02:49:08.626044 29118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.617576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:36289 (local address 127.9.166.1:35757)
0621 02:49:08.618677 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:08.618698 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:08.618894 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:08.619300 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:08.619304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:08.619331 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:08.619761 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:08.619772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.621188 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.621193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:08.623907 (+ 2714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:08.623915 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.625232 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.625238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:08.625443 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:08.625476 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 02:49:08.625562 (+ 86us) client_negotiation.cc:240] Negotiation successful
0621 02:49:08.625689 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":166,"thread_start_us":800,"threads_started":1}
I20260621 02:49:08.626631 29119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.617661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:35121 (local address 127.9.166.1:49213)
0621 02:49:08.619000 (+ 1339us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:08.619014 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:08.619156 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:08.619555 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:08.619559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:08.619579 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:08.620029 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:08.620041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.621345 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.621350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:08.623898 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:08.623913 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.625313 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.625319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:08.625520 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:08.625552 (+ 32us) client_negotiation.cc:769] Sending connection context
0621 02:49:08.626196 (+ 644us) client_negotiation.cc:240] Negotiation successful
0621 02:49:08.626316 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1213,"mutex_wait_us":172,"thread_start_us":990,"threads_started":1}
I20260621 02:49:08.627033 29046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "067a0e8b35374d08b43bff957747ffdf" candidate_uuid: "ea5efec512bd499e8c1640b8673fef62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9535d52f2ad6471392f6400349736ea7" is_pre_election: true
I20260621 02:49:08.627566 29046 raft_consensus.cc:2454] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea5efec512bd499e8c1640b8673fef62 in term 0.
I20260621 02:49:08.627565 28916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "067a0e8b35374d08b43bff957747ffdf" candidate_uuid: "ea5efec512bd499e8c1640b8673fef62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25b5cd734c6942f6837716094e09f3d1" is_pre_election: true
I20260621 02:49:08.628057 28916 raft_consensus.cc:2454] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea5efec512bd499e8c1640b8673fef62 in term 0.
I20260621 02:49:08.628163 28719 leader_election.cc:302] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9535d52f2ad6471392f6400349736ea7, ea5efec512bd499e8c1640b8673fef62; no voters:
I20260621 02:49:08.628587 29114 raft_consensus.cc:2781] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:49:08.628734 29114 raft_consensus.cc:482] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:49:08.628863 29114 raft_consensus.cc:3037] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:08.630952 29114 raft_consensus.cc:504] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25b5cd734c6942f6837716094e09f3d1" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 35121 } } peers { permanent_uuid: "ea5efec512bd499e8c1640b8673fef62" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 33623 } } peers { permanent_uuid: "9535d52f2ad6471392f6400349736ea7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 36289 } }
I20260621 02:49:08.631631 29114 leader_election.cc:288] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [CANDIDATE]: Term 1 election: Requested vote from peers 25b5cd734c6942f6837716094e09f3d1 (127.9.166.2:35121), 9535d52f2ad6471392f6400349736ea7 (127.9.166.3:36289)
I20260621 02:49:08.632025 28916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "067a0e8b35374d08b43bff957747ffdf" candidate_uuid: "ea5efec512bd499e8c1640b8673fef62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25b5cd734c6942f6837716094e09f3d1"
I20260621 02:49:08.632102 29046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "067a0e8b35374d08b43bff957747ffdf" candidate_uuid: "ea5efec512bd499e8c1640b8673fef62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9535d52f2ad6471392f6400349736ea7"
I20260621 02:49:08.632218 28916 raft_consensus.cc:3037] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:08.632298 29046 raft_consensus.cc:3037] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:08.634410 29046 raft_consensus.cc:2454] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea5efec512bd499e8c1640b8673fef62 in term 1.
I20260621 02:49:08.634420 28916 raft_consensus.cc:2454] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea5efec512bd499e8c1640b8673fef62 in term 1.
I20260621 02:49:08.634776 28719 leader_election.cc:302] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9535d52f2ad6471392f6400349736ea7, ea5efec512bd499e8c1640b8673fef62; no voters:
I20260621 02:49:08.635087 29114 raft_consensus.cc:2781] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:08.635566 29114 raft_consensus.cc:686] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [term 1 LEADER]: Becoming Leader. State: Replica: ea5efec512bd499e8c1640b8673fef62, State: Running, Role: LEADER
I20260621 02:49:08.636034 29114 consensus_queue.cc:237] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25b5cd734c6942f6837716094e09f3d1" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 35121 } } peers { permanent_uuid: "ea5efec512bd499e8c1640b8673fef62" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 33623 } } peers { permanent_uuid: "9535d52f2ad6471392f6400349736ea7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 36289 } }
I20260621 02:49:08.641805 28643 catalog_manager.cc:5495] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea5efec512bd499e8c1640b8673fef62 (127.9.166.1). New cstate: current_term: 1 leader_uuid: "ea5efec512bd499e8c1640b8673fef62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25b5cd734c6942f6837716094e09f3d1" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 35121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea5efec512bd499e8c1640b8673fef62" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 33623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9535d52f2ad6471392f6400349736ea7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 36289 } health_report { overall_health: UNKNOWN } } }
I20260621 02:49:08.681244 29106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.676091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48794 (local address 127.9.166.1:33623)
0621 02:49:08.676265 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:08.676270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:08.676287 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:49:08.676411 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:08.676414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:08.676508 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:08.676636 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:49:08.677032 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.677732 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.679151 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.679645 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.679741 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:08.680247 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:49:08.680729 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:49:08.680866 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:08.681004 (+ 138us) server_negotiation.cc:299] Negotiation successful
0621 02:49:08.681041 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 02:49:08.689373 28916 raft_consensus.cc:1261] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1 [term 1 FOLLOWER]: Refusing update from remote peer ea5efec512bd499e8c1640b8673fef62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:49:08.689386 29046 raft_consensus.cc:1261] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7 [term 1 FOLLOWER]: Refusing update from remote peer ea5efec512bd499e8c1640b8673fef62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:49:08.690358 29120 consensus_queue.cc:1035] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b5cd734c6942f6837716094e09f3d1" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:49:08.690739 29114 consensus_queue.cc:1035] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9535d52f2ad6471392f6400349736ea7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 36289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:49:08.717340 28643 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:49:08.717558 28643 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:49:08.721091 28643 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d07d57560e643008e3c8a5701f7ac38: Sending DeleteTablet for 3 replicas of tablet 067a0e8b35374d08b43bff957747ffdf
I20260621 02:49:08.722321 28896 tablet_service.cc:1514] Processing DeleteTablet for tablet 067a0e8b35374d08b43bff957747ffdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:49:08 UTC) from {username='slave'} at 127.0.0.1:52978
I20260621 02:49:08.722364 28765 tablet_service.cc:1514] Processing DeleteTablet for tablet 067a0e8b35374d08b43bff957747ffdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:49:08 UTC) from {username='slave'} at 127.0.0.1:48792
I20260621 02:49:08.722523 29026 tablet_service.cc:1514] Processing DeleteTablet for tablet 067a0e8b35374d08b43bff957747ffdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:49:08 UTC) from {username='slave'} at 127.0.0.1:35162
I20260621 02:49:08.725207 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 28703
I20260621 02:49:08.737530 29133 tablet_replica.cc:330] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1: stopping tablet replica
I20260621 02:49:08.738004 29133 raft_consensus.cc:2229] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:49:08.738538 29133 raft_consensus.cc:2258] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:49:08.740909 29133 ts_tablet_manager.cc:1905] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:49:08.747985 29133 ts_tablet_manager.cc:1918] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:49:08.748140 29133 log.cc:1196] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/067a0e8b35374d08b43bff957747ffdf
I20260621 02:49:08.748709 29133 ts_tablet_manager.cc:1939] T 067a0e8b35374d08b43bff957747ffdf P 25b5cd734c6942f6837716094e09f3d1: Deleting consensus metadata
I20260621 02:49:08.750653 28631 catalog_manager.cc:4841] TS 25b5cd734c6942f6837716094e09f3d1 (127.9.166.2:35121): tablet 067a0e8b35374d08b43bff957747ffdf (table test-table [id=14c464fc86a04ce6bbaecb5e34d587d5]) successfully deleted
I20260621 02:49:08.752646 29135 tablet_replica.cc:330] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7: stopping tablet replica
I20260621 02:49:08.753093 29135 raft_consensus.cc:2229] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:49:08.753616 29135 raft_consensus.cc:2258] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:49:08.755743 29135 ts_tablet_manager.cc:1905] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:49:08.760995 29134 tablet_replica.cc:330] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62: stopping tablet replica
I20260621 02:49:08.761449 29134 raft_consensus.cc:2229] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:49:08.762207 29134 raft_consensus.cc:2258] T 067a0e8b35374d08b43bff957747ffdf P ea5efec512bd499e8c1640b8673fef62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:49:08.762657 29106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.726988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48810 (local address 127.9.166.1:33623)
0621 02:49:08.727222 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:08.727231 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:08.751972 (+ 24741us) server_negotiation.cc:407] Connection header received
0621 02:49:08.752170 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:08.752179 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:08.752323 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:08.752466 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:08.753575 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.755173 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.757919 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.758863 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.758992 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:08.761041 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:08.761050 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:08.761086 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:08.761787 (+ 701us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:08.761835 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:08.761861 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:08.761979 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:08.762131 (+ 152us) server_negotiation.cc:299] Negotiation successful
0621 02:49:08.762232 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:49:08.763665 28825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:08.764873 29135 ts_tablet_manager.cc:1918] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:49:08.765041 29135 log.cc:1196] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/067a0e8b35374d08b43bff957747ffdf
I20260621 02:49:08.765599 29135 ts_tablet_manager.cc:1939] T 067a0e8b35374d08b43bff957747ffdf P 9535d52f2ad6471392f6400349736ea7: Deleting consensus metadata
I20260621 02:49:08.767552 28629 catalog_manager.cc:4841] TS 9535d52f2ad6471392f6400349736ea7 (127.9.166.3:36289): tablet 067a0e8b35374d08b43bff957747ffdf (table test-table [id=14c464fc86a04ce6bbaecb5e34d587d5]) successfully deleted
I20260621 02:49:08.861183 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 28703
W20260621 02:49:08.879365 28629 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.166.1:33623 (error 108)
I20260621 02:49:08.880108 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 28834
I20260621 02:49:08.881340 29103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.880844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:33623 (local address 127.0.0.1:48814)
0621 02:49:08.881097 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:08.881191 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:33623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260621 02:49:08.881614 28629 catalog_manager.cc:4584] TS ea5efec512bd499e8c1640b8673fef62 (127.9.166.1:33623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 067a0e8b35374d08b43bff957747ffdf: Network error: Client connection negotiation failed: client connection to 127.9.166.1:33623: connect: Connection refused (error 111)
I20260621 02:49:08.887115 29105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.880845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52994 (local address 127.9.166.2:35121)
0621 02:49:08.881113 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:08.881119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:08.881139 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:49:08.881280 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:08.881286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:08.881388 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:08.881496 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:08.881909 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.882662 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.884121 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:08.884536 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:08.884670 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:08.885757 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:08.885773 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:08.885786 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:08.886354 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:08.886399 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:08.886423 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:08.886556 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:08.886692 (+ 136us) server_negotiation.cc:299] Negotiation successful
0621 02:49:08.886818 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260621 02:49:08.887789 28956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:08.935433 29103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.934915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:33623 (local address 127.0.0.1:48828)
0621 02:49:08.935124 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:08.935226 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:33623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260621 02:49:08.936465 29103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.936128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:33623 (local address 127.0.0.1:48830)
0621 02:49:08.936252 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:08.936319 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:33623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260621 02:49:08.988382 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 28834
I20260621 02:49:08.997495 29103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.997052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:33623 (local address 127.0.0.1:48840)
0621 02:49:08.997255 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:08.997347 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:33623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 02:49:08.998683 29103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:08.998250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:33623 (local address 127.0.0.1:48854)
0621 02:49:08.998401 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:08.998485 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:33623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260621 02:49:09.009539 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 28964
I20260621 02:49:09.016080 29108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:09.010274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35176 (local address 127.9.166.3:36289)
0621 02:49:09.010436 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:09.010440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:09.010458 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:49:09.010591 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:09.010595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:09.010679 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:09.010761 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:09.011218 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:09.012049 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:09.013321 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:09.013691 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:09.013778 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:09.014796 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:09.014803 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:09.014817 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:09.015482 (+ 665us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:09.015526 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:09.015546 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:09.015628 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:09.015759 (+ 131us) server_negotiation.cc:299] Negotiation successful
0621 02:49:09.015843 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 02:49:09.016700 29086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:09.079579 29103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:09.079040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:33623 (local address 127.0.0.1:48868)
0621 02:49:09.079264 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:09.079400 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:33623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260621 02:49:09.080706 29103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:09.080256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:33623 (local address 127.0.0.1:48880)
0621 02:49:09.080476 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:09.080572 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:33623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260621 02:49:09.105027 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 28964
I20260621 02:49:09.121826 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 28613
I20260621 02:49:09.122977 28674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:09.233184 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 28613
2026-06-21T02:49:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2713 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:09 dist-test-slave-hvgn krb5kdc[29144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:09 dist-test-slave-hvgn krb5kdc[29144](info): set up 2 sockets
Jun 21 02:49:09 dist-test-slave-hvgn krb5kdc[29144](info): commencing operation
krb5kdc: starting...
W20260621 02:49:10.339633 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:10 dist-test-slave-hvgn krb5kdc[29144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:49:10.590126 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:46123
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:46123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:10.917730 29157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:10.918013 29157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:10.918218 29157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:10.925482 29157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:10.925586 29157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:10.925688 29157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:10.925787 29157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:10.925868 29157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:10.935953 29157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:46123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:46123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:10.937290 29157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:10.938640 29157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:10.950429 29163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:10.950342 29165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:10.950392 29162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:10.951431 29157 server_base.cc:1034] running on GCE node
I20260621 02:49:10.952175 29157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:10.953584 29157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:10.954821 29157 hybrid_clock.cc:648] HybridClock initialized: now 1782010150954748 us; error 56 us; skew 500 ppm
I20260621 02:49:10.955127 29157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:10 dist-test-slave-hvgn krb5kdc[29144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010150, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:10.963238 29157 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:49:10.964743 29157 webserver.cc:460] Webserver started at http://127.9.166.62:46439/ using document root <none> and password file <none>
I20260621 02:49:10.965387 29157 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:10.965482 29157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:10.965721 29157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:10.968653 29157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "4c9f534fad6f490e8c184dad007c75bf"
format_stamp: "Formatted at 2026-06-21 02:49:10 on dist-test-slave-hvgn"
server_key: "1c0407ee158b88cbc32c8d8e4a16f6f1"
server_key_iv: "f9cd27121097f1b3618d00fac0f53d99"
server_key_version: "encryptionkey@0"
I20260621 02:49:10.969353 29157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "4c9f534fad6f490e8c184dad007c75bf"
format_stamp: "Formatted at 2026-06-21 02:49:10 on dist-test-slave-hvgn"
server_key: "1c0407ee158b88cbc32c8d8e4a16f6f1"
server_key_iv: "f9cd27121097f1b3618d00fac0f53d99"
server_key_version: "encryptionkey@0"
I20260621 02:49:10.974819 29157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260621 02:49:10.978221 29172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:10.979795 29157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:49:10.979980 29157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "4c9f534fad6f490e8c184dad007c75bf"
format_stamp: "Formatted at 2026-06-21 02:49:10 on dist-test-slave-hvgn"
server_key: "1c0407ee158b88cbc32c8d8e4a16f6f1"
server_key_iv: "f9cd27121097f1b3618d00fac0f53d99"
server_key_version: "encryptionkey@0"
I20260621 02:49:10.980162 29157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:10.999577 29157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:11.003810 29157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:11.004122 29157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:11.022818 29157 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:46123
I20260621 02:49:11.022847 29223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:46123 every 8 connection(s)
I20260621 02:49:11.024298 29157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:11.028003 29224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:11.034085 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 29157
I20260621 02:49:11.034356 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:11.034832 9880 external_mini_cluster.cc:1428] Setting key 362e2dc43fa1a2e1e906a7a4603cdcdb
I20260621 02:49:11.039331 29224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf: Bootstrap starting.
I20260621 02:49:11.043193 29224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:11.044448 29224 log.cc:824] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:11.047621 29224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf: No bootstrap required, opened a new log
Jun 21 02:49:11 dist-test-slave-hvgn krb5kdc[29144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:49:11.056991 29224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9f534fad6f490e8c184dad007c75bf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46123 } }
I20260621 02:49:11.057456 29224 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:11.057565 29224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c9f534fad6f490e8c184dad007c75bf, State: Initialized, Role: FOLLOWER
I20260621 02:49:11.058244 29224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9f534fad6f490e8c184dad007c75bf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46123 } }
I20260621 02:49:11.058534 29224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:11.058668 29224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:11.058857 29224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:11.060312 29227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:11.036868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57296 (local address 127.9.166.62:46123)
0621 02:49:11.037636 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:11.037651 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:11.037701 (+ 50us) server_negotiation.cc:407] Connection header received
0621 02:49:11.040105 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:11.040132 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:11.040622 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:11.041075 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:11.042428 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:11.044484 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:11.046207 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:11.047223 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:11.047431 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:11.051303 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:11.051312 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:11.051348 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:11.051428 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:11.055578 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:11.056291 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:11.056299 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:11.056307 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:11.056505 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:11.056955 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:11.056960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:11.056964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:11.057723 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:11.058040 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:11.058480 (+ 440us) server_negotiation.cc:299] Negotiation successful
0621 02:49:11.058935 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":326,"threads_started":1}
I20260621 02:49:11.062059 29224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9f534fad6f490e8c184dad007c75bf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46123 } }
I20260621 02:49:11.062548 29224 leader_election.cc:302] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c9f534fad6f490e8c184dad007c75bf; no voters:
I20260621 02:49:11.063176 29224 leader_election.cc:288] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:11.063434 29229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:11.064123 29229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf [term 1 LEADER]: Becoming Leader. State: Replica: 4c9f534fad6f490e8c184dad007c75bf, State: Running, Role: LEADER
I20260621 02:49:11.064603 29229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9f534fad6f490e8c184dad007c75bf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46123 } }
I20260621 02:49:11.065172 29224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:11.067281 29231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c9f534fad6f490e8c184dad007c75bf. Latest consensus state: current_term: 1 leader_uuid: "4c9f534fad6f490e8c184dad007c75bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9f534fad6f490e8c184dad007c75bf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46123 } } }
I20260621 02:49:11.067307 29230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c9f534fad6f490e8c184dad007c75bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9f534fad6f490e8c184dad007c75bf" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 46123 } } }
I20260621 02:49:11.067703 29231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:11.067807 29230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:11.072455 29235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:11.077978 29235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:11.086702 29235 catalog_manager.cc:1347] Generated new cluster ID: bc960cec697a4e4188703bbc15cbed18
I20260621 02:49:11.086817 29235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:11.097507 29235 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:11.098791 29235 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:11.104955 29235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf: Generated new TSK 0
I20260621 02:49:11.105871 29235 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.166.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
Entry for principal kudu/127.9.166.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab.
I20260621 02:49:11.172176 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:46123
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:11.523028 29250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:11.523293 29250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:11.523481 29250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:11.531014 29250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:11.531114 29250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:11.531335 29250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:49:11.541119 29250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.1.keytab
--principal=kudu/127.9.166.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:46123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:11.542294 29250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:11.543637 29250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:11.556128 29255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:11.556190 29256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:11.556356 29258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:11.557314 29250 server_base.cc:1034] running on GCE node
I20260621 02:49:11.557907 29250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:11.558790 29250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:11.560041 29250 hybrid_clock.cc:648] HybridClock initialized: now 1782010151559990 us; error 73 us; skew 500 ppm
I20260621 02:49:11.560453 29250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:11 dist-test-slave-hvgn krb5kdc[29144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010151, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:11.563735 29250 init.cc:377] Logged in from keytab as kudu/127.9.166.1@KRBTEST.COM (short username kudu)
I20260621 02:49:11.565100 29250 webserver.cc:460] Webserver started at http://127.9.166.1:40289/ using document root <none> and password file <none>
I20260621 02:49:11.565825 29250 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:11.565922 29250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:11.566182 29250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:11.569293 29250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "601593c14778426fb4ca72255886cb23"
format_stamp: "Formatted at 2026-06-21 02:49:11 on dist-test-slave-hvgn"
server_key: "ef1bf005d17d9a22ec6cf67621a9ce37"
server_key_iv: "10414ad0d622a782d2a8a1e17fcba2eb"
server_key_version: "encryptionkey@0"
I20260621 02:49:11.569993 29250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "601593c14778426fb4ca72255886cb23"
format_stamp: "Formatted at 2026-06-21 02:49:11 on dist-test-slave-hvgn"
server_key: "ef1bf005d17d9a22ec6cf67621a9ce37"
server_key_iv: "10414ad0d622a782d2a8a1e17fcba2eb"
server_key_version: "encryptionkey@0"
I20260621 02:49:11.574976 29250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 02:49:11.578373 29265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:11.579809 29250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260621 02:49:11.579932 29250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "601593c14778426fb4ca72255886cb23"
format_stamp: "Formatted at 2026-06-21 02:49:11 on dist-test-slave-hvgn"
server_key: "ef1bf005d17d9a22ec6cf67621a9ce37"
server_key_iv: "10414ad0d622a782d2a8a1e17fcba2eb"
server_key_version: "encryptionkey@0"
I20260621 02:49:11.580104 29250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:11.603883 29250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:11.608552 29250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:11.608842 29250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:11.609754 29250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:49:11.611016 29250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:49:11.611088 29250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:11.611207 29250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:49:11.611277 29250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:11.643875 29250 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:34113
I20260621 02:49:11.643992 29377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:34113 every 8 connection(s)
I20260621 02:49:11.645215 29250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:49:11.648841 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 29250
I20260621 02:49:11.649070 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:49:11.649626 9880 external_mini_cluster.cc:1428] Setting key c531da2ffb57b008c646dc5c0b83e41d
Jun 21 02:49:11 dist-test-slave-hvgn krb5kdc[29144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010151, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.1@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:49:11.668682 29380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:11.647920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:44053 (local address 127.9.166.62:46123)
0621 02:49:11.648415 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:11.648420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:11.649857 (+ 1437us) server_negotiation.cc:407] Connection header received
0621 02:49:11.652894 (+ 3037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:11.652901 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:11.653062 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:11.653199 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:11.654902 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:11.655702 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:11.657314 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:11.657721 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:11.657858 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:11.661570 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:11.661580 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:11.661589 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:11.661635 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:11.665166 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:11.666007 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:11.666015 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:11.666022 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:11.666170 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:11.666619 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:11.666627 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:11.666633 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:11.667041 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:11.667193 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:11.667921 (+ 728us) server_negotiation.cc:299] Negotiation successful
0621 02:49:11.668181 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":264,"threads_started":1}
I20260621 02:49:11.669298 29381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:11.648571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:46123 (local address 127.9.166.1:44053)
0621 02:49:11.649653 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:11.649735 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:11.652508 (+ 2773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:11.653553 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:11.653586 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:11.654140 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:11.654719 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:11.654737 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:11.655857 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:11.655862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:11.657155 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:11.657168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:11.657949 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:11.657956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:11.658214 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:11.658905 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:11.658927 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:11.661349 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:11.665345 (+ 3996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:11.665352 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:11.665364 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:11.665781 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:11.666275 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:11.666278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:11.666280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:11.666504 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:11.667166 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:11.667172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:11.667531 (+ 359us) client_negotiation.cc:769] Sending connection context
0621 02:49:11.667770 (+ 239us) client_negotiation.cc:240] Negotiation successful
0621 02:49:11.668174 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":1175,"threads_started":1}
I20260621 02:49:11.670964 29378 heartbeater.cc:347] Connected to a master server at 127.9.166.62:46123
I20260621 02:49:11.671310 29378 heartbeater.cc:464] Registering TS with master...
I20260621 02:49:11.672255 29378 heartbeater.cc:511] Master 127.9.166.62:46123 requested a full tablet report, sending...
I20260621 02:49:11.674651 29189 ts_manager.cc:194] Registered new tserver with Master: 601593c14778426fb4ca72255886cb23 (127.9.166.1:34113)
WARNING: no policy specified for kudu/127.9.166.2@KRBTEST.COM; defaulting to no policy
I20260621 02:49:11.676178 29189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.1@KRBTEST.COM'} at 127.9.166.1:44053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
Entry for principal kudu/127.9.166.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab.
I20260621 02:49:11.704706 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:46123
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:12.026463 29384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:12.026741 29384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:12.026894 29384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:12.034274 29384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:12.034374 29384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:12.034575 29384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:49:12.044257 29384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.2.keytab
--principal=kudu/127.9.166.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:46123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:12.045471 29384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:12.046790 29384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:12.059544 29389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:12.059605 29390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:12.059659 29392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:12.060456 29384 server_base.cc:1034] running on GCE node
I20260621 02:49:12.060956 29384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:12.061780 29384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:12.062973 29384 hybrid_clock.cc:648] HybridClock initialized: now 1782010152062910 us; error 49 us; skew 500 ppm
I20260621 02:49:12.063282 29384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:12 dist-test-slave-hvgn krb5kdc[29144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010152, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:12.066298 29384 init.cc:377] Logged in from keytab as kudu/127.9.166.2@KRBTEST.COM (short username kudu)
I20260621 02:49:12.067921 29384 webserver.cc:460] Webserver started at http://127.9.166.2:41691/ using document root <none> and password file <none>
I20260621 02:49:12.068576 29384 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:12.068701 29384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:12.069024 29384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:12.071857 29384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "3f2b622a4996405c92db15ce1ea58685"
format_stamp: "Formatted at 2026-06-21 02:49:12 on dist-test-slave-hvgn"
server_key: "7d37c3536d3c7617709aa97c8f7d82e6"
server_key_iv: "212ab4eb5edc87c7eaf120da4237ef69"
server_key_version: "encryptionkey@0"
I20260621 02:49:12.072633 29384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "3f2b622a4996405c92db15ce1ea58685"
format_stamp: "Formatted at 2026-06-21 02:49:12 on dist-test-slave-hvgn"
server_key: "7d37c3536d3c7617709aa97c8f7d82e6"
server_key_iv: "212ab4eb5edc87c7eaf120da4237ef69"
server_key_version: "encryptionkey@0"
I20260621 02:49:12.077504 29384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:49:12.080802 29399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:12.082121 29384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:49:12.082311 29384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "3f2b622a4996405c92db15ce1ea58685"
format_stamp: "Formatted at 2026-06-21 02:49:12 on dist-test-slave-hvgn"
server_key: "7d37c3536d3c7617709aa97c8f7d82e6"
server_key_iv: "212ab4eb5edc87c7eaf120da4237ef69"
server_key_version: "encryptionkey@0"
I20260621 02:49:12.082474 29384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:12.106010 29384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:12.110428 29384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:12.110728 29384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:12.111755 29384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:49:12.113114 29384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:49:12.113185 29384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:12.113356 29384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:49:12.113408 29384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:12.146742 29384 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:46775
I20260621 02:49:12.146852 29511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:46775 every 8 connection(s)
I20260621 02:49:12.148176 29384 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:49:12.157384 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 29384
I20260621 02:49:12.157567 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:49:12.158020 9880 external_mini_cluster.cc:1428] Setting key 571de97947165c3d5ab08356a557a8cc
Jun 21 02:49:12 dist-test-slave-hvgn krb5kdc[29144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010152, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.2@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:49:12.170814 29380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:12.150828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:55039 (local address 127.9.166.62:46123)
0621 02:49:12.151036 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:12.151042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:12.152395 (+ 1353us) server_negotiation.cc:407] Connection header received
0621 02:49:12.154429 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:12.154436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:12.154536 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:12.154657 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:12.156313 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.157111 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.159090 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.159707 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.159847 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:12.163235 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:12.163245 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:12.163253 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:12.163302 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:12.167569 (+ 4267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:12.168309 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:12.168318 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:12.168326 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:12.168511 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:12.168933 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:12.168941 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:12.168948 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:12.169315 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:12.169439 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:12.170398 (+ 959us) server_negotiation.cc:299] Negotiation successful
0621 02:49:12.170568 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260621 02:49:12.172334 29514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:12.151488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:46123 (local address 127.9.166.2:55039)
0621 02:49:12.152218 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:12.152263 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:12.154136 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:12.154931 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:12.154940 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:12.155519 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:12.156121 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:12.156139 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.157274 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.157280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:12.158920 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:12.158935 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.159948 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.159956 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:12.160210 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:12.160899 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:12.160920 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:12.163000 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:12.167732 (+ 4732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:12.167739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:12.167752 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:12.168184 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:12.168626 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:12.168629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:12.168631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:12.168814 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:12.169452 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:12.169460 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:12.169964 (+ 504us) client_negotiation.cc:769] Sending connection context
0621 02:49:12.170297 (+ 333us) client_negotiation.cc:240] Negotiation successful
0621 02:49:12.170864 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":282,"threads_started":1}
I20260621 02:49:12.173817 29512 heartbeater.cc:347] Connected to a master server at 127.9.166.62:46123
I20260621 02:49:12.174124 29512 heartbeater.cc:464] Registering TS with master...
I20260621 02:49:12.174801 29512 heartbeater.cc:511] Master 127.9.166.62:46123 requested a full tablet report, sending...
I20260621 02:49:12.176302 29189 ts_manager.cc:194] Registered new tserver with Master: 3f2b622a4996405c92db15ce1ea58685 (127.9.166.2:46775)
I20260621 02:49:12.177531 29189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.2@KRBTEST.COM'} at 127.9.166.2:55039
WARNING: no policy specified for kudu/127.9.166.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
Entry for principal kudu/127.9.166.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab.
I20260621 02:49:12.213985 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:46123
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:37283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:12.543375 29517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:12.543753 29517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:12.544152 29517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:12.551544 29517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:12.551713 29517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:12.551918 29517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:49:12.561713 29517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:37283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.3.keytab
--principal=kudu/127.9.166.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:46123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:12.563073 29517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:12.564404 29517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:12.577847 29523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:12.577920 29525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:12.578424 29522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:12.578557 29517 server_base.cc:1034] running on GCE node
I20260621 02:49:12.579141 29517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:12.580180 29517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:12.581408 29517 hybrid_clock.cc:648] HybridClock initialized: now 1782010152581348 us; error 51 us; skew 500 ppm
I20260621 02:49:12.581704 29517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:12 dist-test-slave-hvgn krb5kdc[29144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010152, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:12.585156 29517 init.cc:377] Logged in from keytab as kudu/127.9.166.3@KRBTEST.COM (short username kudu)
I20260621 02:49:12.586752 29517 webserver.cc:460] Webserver started at http://127.9.166.3:37215/ using document root <none> and password file <none>
I20260621 02:49:12.587486 29517 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:12.587615 29517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:12.587930 29517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:12.590708 29517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "efd7603353df412195c2934295062879"
format_stamp: "Formatted at 2026-06-21 02:49:12 on dist-test-slave-hvgn"
server_key: "2deb9a7cd400809acb3056d46e210a29"
server_key_iv: "cf01c8d890950ea041fe7ad4579f3e48"
server_key_version: "encryptionkey@0"
I20260621 02:49:12.591594 29517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "efd7603353df412195c2934295062879"
format_stamp: "Formatted at 2026-06-21 02:49:12 on dist-test-slave-hvgn"
server_key: "2deb9a7cd400809acb3056d46e210a29"
server_key_iv: "cf01c8d890950ea041fe7ad4579f3e48"
server_key_version: "encryptionkey@0"
I20260621 02:49:12.596644 29517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:49:12.600241 29532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:12.601609 29517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:49:12.601831 29517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "efd7603353df412195c2934295062879"
format_stamp: "Formatted at 2026-06-21 02:49:12 on dist-test-slave-hvgn"
server_key: "2deb9a7cd400809acb3056d46e210a29"
server_key_iv: "cf01c8d890950ea041fe7ad4579f3e48"
server_key_version: "encryptionkey@0"
I20260621 02:49:12.602046 29517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:12.625942 29517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:12.630192 29517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:12.630507 29517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:12.631569 29517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:49:12.632916 29517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:49:12.633023 29517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:12.633175 29517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:49:12.633260 29517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:12.675144 29517 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:44701
I20260621 02:49:12.675232 29644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:44701 every 8 connection(s)
I20260621 02:49:12.676514 29517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:49:12.677949 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 29517
I20260621 02:49:12.678143 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:49:12.678681 9880 external_mini_cluster.cc:1428] Setting key 07c1b056fe2aaab0e11a7cfe440b2003
I20260621 02:49:12.679255 29378 heartbeater.cc:503] Master 127.9.166.62:46123 was elected leader, sending a full tablet report...
Jun 21 02:49:12 dist-test-slave-hvgn krb5kdc[29144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010152, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.3@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:49:12.697661 29647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:12.679307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:53275 (local address 127.9.166.62:46123)
0621 02:49:12.680183 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:12.680188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:12.680945 (+ 757us) server_negotiation.cc:407] Connection header received
0621 02:49:12.683028 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:12.683033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:12.683126 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:12.683225 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:12.685084 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.685929 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.687740 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.688160 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.688296 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:12.691695 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:12.691700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:12.691703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:12.691734 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:12.694678 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:12.695379 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:12.695433 (+ 54us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:12.695437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:12.695561 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:12.695978 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:12.695983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:12.695986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:12.696247 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:12.696403 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:12.697082 (+ 679us) server_negotiation.cc:299] Negotiation successful
0621 02:49:12.697295 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":642,"threads_started":1}
I20260621 02:49:12.698593 29648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:12.679922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:46123 (local address 127.9.166.3:53275)
0621 02:49:12.680800 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:12.680841 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:12.682773 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:12.683527 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:12.683538 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:12.684216 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:12.684903 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:12.684942 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.686082 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.686087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:12.687598 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:12.687610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.688386 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.688394 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:12.688626 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:12.689309 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:12.689331 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:12.691491 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:12.694804 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:12.694814 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:12.694831 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:12.695242 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:12.695670 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:12.695675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:12.695678 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:12.695861 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:12.696380 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:12.696388 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:12.696721 (+ 333us) client_negotiation.cc:769] Sending connection context
0621 02:49:12.696978 (+ 257us) client_negotiation.cc:240] Negotiation successful
0621 02:49:12.697500 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":421,"threads_started":1}
I20260621 02:49:12.699965 29645 heartbeater.cc:347] Connected to a master server at 127.9.166.62:46123
I20260621 02:49:12.700294 29645 heartbeater.cc:464] Registering TS with master...
I20260621 02:49:12.700974 29645 heartbeater.cc:511] Master 127.9.166.62:46123 requested a full tablet report, sending...
I20260621 02:49:12.702386 29188 ts_manager.cc:194] Registered new tserver with Master: efd7603353df412195c2934295062879 (127.9.166.3:44701)
I20260621 02:49:12.703248 29188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.166.3@KRBTEST.COM'} at 127.9.166.3:53275
I20260621 02:49:12.706709 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:49:12.723552 29647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:12.711036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52426 (local address 127.9.166.62:46123)
0621 02:49:12.711201 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:12.711206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:12.711702 (+ 496us) server_negotiation.cc:407] Connection header received
0621 02:49:12.711859 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:12.711863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:12.711925 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:12.712007 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:12.713417 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.714142 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.715474 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.715890 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.715986 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:12.718410 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:12.718415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:12.718419 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:12.718450 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:12.721111 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:12.721764 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:12.721768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:12.721770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:12.721911 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:12.722279 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:12.722283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:12.722286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:12.722587 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:12.722758 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:12.723098 (+ 340us) server_negotiation.cc:299] Negotiation successful
0621 02:49:12.723258 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:49:12.727555 29188 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:49:12.734397 29188 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:49:12.772166 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:12.747013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:34113 (local address 127.0.0.1:38848)
0621 02:49:12.759805 (+ 12792us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:12.759875 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:12.760130 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:12.764461 (+ 4331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:12.764469 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:12.764517 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:12.765097 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:12.765112 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.767328 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.767333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:12.769283 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:12.769295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.771103 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.771109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:12.771315 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:12.771367 (+ 52us) client_negotiation.cc:769] Sending connection context
0621 02:49:12.771525 (+ 158us) client_negotiation.cc:240] Negotiation successful
0621 02:49:12.771703 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12418,"mutex_wait_us":4,"thread_start_us":15604,"threads_started":1}
I20260621 02:49:12.772311 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:12.747104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:46775 (local address 127.0.0.1:42426)
0621 02:49:12.762346 (+ 15242us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:12.762370 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:12.762522 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:12.763876 (+ 1354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:12.763887 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:12.763929 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:12.764453 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:12.764470 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.766888 (+ 2418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.766893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:12.769455 (+ 2562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:12.769471 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.771523 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.771531 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:12.771702 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:12.771735 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 02:49:12.771812 (+ 77us) client_negotiation.cc:240] Negotiation successful
0621 02:49:12.771929 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15081,"thread_start_us":13260,"threads_started":1}
I20260621 02:49:12.772840 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:12.747448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.9.166.2:46775)
0621 02:49:12.762711 (+ 15263us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:12.762725 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:12.762764 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:49:12.762902 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:12.762914 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:12.763245 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:12.763517 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:12.764718 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.766701 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.769614 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.771037 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.771979 (+ 942us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:12.772196 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:12.772267 (+ 71us) server_negotiation.cc:299] Negotiation successful
0621 02:49:12.772446 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15044,"thread_start_us":15353,"threads_started":1}
I20260621 02:49:12.773208 29660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:12.747355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38848 (local address 127.9.166.1:34113)
0621 02:49:12.763253 (+ 15898us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:12.763264 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:12.763295 (+ 31us) server_negotiation.cc:407] Connection header received
0621 02:49:12.763434 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:12.763446 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:12.763746 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:12.763972 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:12.765331 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.767162 (+ 1831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.769434 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.770926 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.771055 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:12.771334 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:12.772440 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0621 02:49:12.772590 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15710,"thread_start_us":16798,"threads_started":1}
I20260621 02:49:12.776966 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:12.761700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.9.166.3:44701)
0621 02:49:12.766510 (+ 4810us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:12.766517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:12.767130 (+ 613us) server_negotiation.cc:407] Connection header received
0621 02:49:12.767229 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:12.767237 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:12.767488 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:12.767991 (+ 503us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:12.768519 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.770000 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.772473 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.775740 (+ 3267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.775867 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:12.776091 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:12.776498 (+ 407us) server_negotiation.cc:299] Negotiation successful
0621 02:49:12.776644 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4644,"thread_start_us":4499,"threads_started":1}
I20260621 02:49:12.777004 29663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:12.763160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:44701 (local address 127.0.0.1:33792)
0621 02:49:12.766815 (+ 3655us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:12.766835 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:12.766982 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:12.767691 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:12.767698 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:12.767728 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:12.768305 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:12.768318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.770155 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.770161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:12.772298 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:12.772318 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:12.776194 (+ 3876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:12.776200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:12.776331 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:12.776352 (+ 21us) client_negotiation.cc:769] Sending connection context
0621 02:49:12.776459 (+ 107us) client_negotiation.cc:240] Negotiation successful
0621 02:49:12.776588 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3497,"mutex_wait_us":51,"thread_start_us":3368,"threads_started":1}
I20260621 02:49:12.776803 29313 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef267b1884142599b46a05befcbc615 (DEFAULT_TABLE table=test-table [id=2660ae142a4c46dd806c6956acdace57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:49:12.776795 29447 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef267b1884142599b46a05befcbc615 (DEFAULT_TABLE table=test-table [id=2660ae142a4c46dd806c6956acdace57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:49:12.778252 29313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef267b1884142599b46a05befcbc615. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:12.778254 29447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef267b1884142599b46a05befcbc615. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:12.780253 29580 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef267b1884142599b46a05befcbc615 (DEFAULT_TABLE table=test-table [id=2660ae142a4c46dd806c6956acdace57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:49:12.781394 29580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef267b1884142599b46a05befcbc615. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:12.792737 29664 tablet_bootstrap.cc:492] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879: Bootstrap starting.
I20260621 02:49:12.794617 29665 tablet_bootstrap.cc:492] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23: Bootstrap starting.
I20260621 02:49:12.796273 29666 tablet_bootstrap.cc:492] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685: Bootstrap starting.
I20260621 02:49:12.796756 29664 tablet_bootstrap.cc:654] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:12.798045 29665 tablet_bootstrap.cc:654] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:12.798188 29664 log.cc:824] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:12.799249 29665 log.cc:824] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:12.799495 29666 tablet_bootstrap.cc:654] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:12.800674 29666 log.cc:824] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:12.801448 29664 tablet_bootstrap.cc:492] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879: No bootstrap required, opened a new log
I20260621 02:49:12.801745 29664 ts_tablet_manager.cc:1397] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260621 02:49:12.802204 29665 tablet_bootstrap.cc:492] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23: No bootstrap required, opened a new log
I20260621 02:49:12.802462 29665 ts_tablet_manager.cc:1397] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260621 02:49:12.803149 29666 tablet_bootstrap.cc:492] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685: No bootstrap required, opened a new log
I20260621 02:49:12.803435 29666 ts_tablet_manager.cc:1397] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260621 02:49:12.811525 29666 raft_consensus.cc:348] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601593c14778426fb4ca72255886cb23" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 34113 } } peers { permanent_uuid: "3f2b622a4996405c92db15ce1ea58685" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 46775 } } peers { permanent_uuid: "efd7603353df412195c2934295062879" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 44701 } }
I20260621 02:49:12.811924 29666 raft_consensus.cc:374] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:12.812019 29666 raft_consensus.cc:729] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f2b622a4996405c92db15ce1ea58685, State: Initialized, Role: FOLLOWER
I20260621 02:49:12.812645 29666 consensus_queue.cc:260] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601593c14778426fb4ca72255886cb23" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 34113 } } peers { permanent_uuid: "3f2b622a4996405c92db15ce1ea58685" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 46775 } } peers { permanent_uuid: "efd7603353df412195c2934295062879" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 44701 } }
I20260621 02:49:12.812718 29664 raft_consensus.cc:348] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601593c14778426fb4ca72255886cb23" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 34113 } } peers { permanent_uuid: "3f2b622a4996405c92db15ce1ea58685" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 46775 } } peers { permanent_uuid: "efd7603353df412195c2934295062879" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 44701 } }
I20260621 02:49:12.813145 29664 raft_consensus.cc:374] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:12.813241 29664 raft_consensus.cc:729] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd7603353df412195c2934295062879, State: Initialized, Role: FOLLOWER
I20260621 02:49:12.813086 29665 raft_consensus.cc:348] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601593c14778426fb4ca72255886cb23" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 34113 } } peers { permanent_uuid: "3f2b622a4996405c92db15ce1ea58685" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 46775 } } peers { permanent_uuid: "efd7603353df412195c2934295062879" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 44701 } }
I20260621 02:49:12.813508 29665 raft_consensus.cc:374] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:12.813606 29665 raft_consensus.cc:729] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601593c14778426fb4ca72255886cb23, State: Initialized, Role: FOLLOWER
I20260621 02:49:12.813817 29664 consensus_queue.cc:260] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601593c14778426fb4ca72255886cb23" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 34113 } } peers { permanent_uuid: "3f2b622a4996405c92db15ce1ea58685" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 46775 } } peers { permanent_uuid: "efd7603353df412195c2934295062879" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 44701 } }
I20260621 02:49:12.814147 29512 heartbeater.cc:503] Master 127.9.166.62:46123 was elected leader, sending a full tablet report...
I20260621 02:49:12.814690 29666 ts_tablet_manager.cc:1428] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260621 02:49:12.815006 29645 heartbeater.cc:503] Master 127.9.166.62:46123 was elected leader, sending a full tablet report...
I20260621 02:49:12.815531 29664 ts_tablet_manager.cc:1428] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260621 02:49:12.815828 29665 consensus_queue.cc:260] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601593c14778426fb4ca72255886cb23" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 34113 } } peers { permanent_uuid: "3f2b622a4996405c92db15ce1ea58685" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 46775 } } peers { permanent_uuid: "efd7603353df412195c2934295062879" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 44701 } }
I20260621 02:49:12.817975 29665 ts_tablet_manager.cc:1428] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260621 02:49:12.897641 29379 tablet.cc:2369] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:49:12.900127 29513 tablet.cc:2369] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:49:12.929096 29646 tablet.cc:2369] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:49:13.133620 29671 raft_consensus.cc:482] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:49:13.133939 29671 raft_consensus.cc:504] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601593c14778426fb4ca72255886cb23" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 34113 } } peers { permanent_uuid: "3f2b622a4996405c92db15ce1ea58685" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 46775 } } peers { permanent_uuid: "efd7603353df412195c2934295062879" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 44701 } }
I20260621 02:49:13.135739 29671 leader_election.cc:288] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 601593c14778426fb4ca72255886cb23 (127.9.166.1:34113), 3f2b622a4996405c92db15ce1ea58685 (127.9.166.2:46775)
I20260621 02:49:13.143769 29660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:13.136283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:50843 (local address 127.9.166.1:34113)
0621 02:49:13.136469 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:13.136475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:13.136607 (+ 132us) server_negotiation.cc:407] Connection header received
0621 02:49:13.136876 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:13.136884 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:13.136986 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:13.137099 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:13.137682 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:13.138804 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:13.141473 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:13.142714 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:13.142848 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:13.142952 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:13.143369 (+ 417us) server_negotiation.cc:299] Negotiation successful
0621 02:49:13.143517 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:49:13.143880 29648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:13.136017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:34113 (local address 127.9.166.3:50843)
0621 02:49:13.136504 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:13.136525 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:13.136668 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:13.137127 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:13.137131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:13.137153 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:13.137530 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:13.137544 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:13.138945 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:13.138950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:13.141299 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:13.141313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:13.142894 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:13.142905 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:13.143143 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:13.143225 (+ 82us) client_negotiation.cc:769] Sending connection context
0621 02:49:13.143323 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 02:49:13.143495 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":175,"spinlock_wait_cycles":24320}
I20260621 02:49:13.143923 29673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:13.136080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:46775 (local address 127.9.166.3:47975)
0621 02:49:13.136679 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:13.136755 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:13.136872 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:13.137242 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:13.137246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:13.137275 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:13.137762 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:13.137771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:13.139085 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:13.139089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:13.141298 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:13.141313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:13.142918 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:13.142922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:13.143159 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:13.143232 (+ 73us) client_negotiation.cc:769] Sending connection context
0621 02:49:13.143323 (+ 91us) client_negotiation.cc:240] Negotiation successful
0621 02:49:13.143495 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":149,"thread_start_us":736,"threads_started":1}
I20260621 02:49:13.143767 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:13.136263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:47975 (local address 127.9.166.2:46775)
0621 02:49:13.136469 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:13.136475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:13.136853 (+ 378us) server_negotiation.cc:407] Connection header received
0621 02:49:13.137014 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:13.137032 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:13.137112 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:13.137204 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:13.137908 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:13.138941 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:13.141473 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:13.142702 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:13.142876 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:13.142958 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:13.143369 (+ 411us) server_negotiation.cc:299] Negotiation successful
0621 02:49:13.143517 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:49:13.144975 29333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef267b1884142599b46a05befcbc615" candidate_uuid: "efd7603353df412195c2934295062879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601593c14778426fb4ca72255886cb23" is_pre_election: true
I20260621 02:49:13.144971 29467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef267b1884142599b46a05befcbc615" candidate_uuid: "efd7603353df412195c2934295062879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f2b622a4996405c92db15ce1ea58685" is_pre_election: true
I20260621 02:49:13.145465 29333 raft_consensus.cc:2454] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efd7603353df412195c2934295062879 in term 0.
I20260621 02:49:13.145469 29467 raft_consensus.cc:2454] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efd7603353df412195c2934295062879 in term 0.
I20260621 02:49:13.146106 29534 leader_election.cc:302] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f2b622a4996405c92db15ce1ea58685, efd7603353df412195c2934295062879; no voters:
I20260621 02:49:13.146523 29671 raft_consensus.cc:2781] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:49:13.146689 29671 raft_consensus.cc:482] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:49:13.146831 29671 raft_consensus.cc:3037] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:13.149076 29671 raft_consensus.cc:504] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601593c14778426fb4ca72255886cb23" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 34113 } } peers { permanent_uuid: "3f2b622a4996405c92db15ce1ea58685" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 46775 } } peers { permanent_uuid: "efd7603353df412195c2934295062879" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 44701 } }
I20260621 02:49:13.149744 29671 leader_election.cc:288] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [CANDIDATE]: Term 1 election: Requested vote from peers 601593c14778426fb4ca72255886cb23 (127.9.166.1:34113), 3f2b622a4996405c92db15ce1ea58685 (127.9.166.2:46775)
I20260621 02:49:13.150192 29333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef267b1884142599b46a05befcbc615" candidate_uuid: "efd7603353df412195c2934295062879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601593c14778426fb4ca72255886cb23"
I20260621 02:49:13.150293 29467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef267b1884142599b46a05befcbc615" candidate_uuid: "efd7603353df412195c2934295062879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f2b622a4996405c92db15ce1ea58685"
I20260621 02:49:13.150398 29333 raft_consensus.cc:3037] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:13.150493 29467 raft_consensus.cc:3037] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:13.152714 29333 raft_consensus.cc:2454] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efd7603353df412195c2934295062879 in term 1.
I20260621 02:49:13.153147 29536 leader_election.cc:302] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 601593c14778426fb4ca72255886cb23, efd7603353df412195c2934295062879; no voters:
I20260621 02:49:13.153465 29671 raft_consensus.cc:2781] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:13.153991 29671 raft_consensus.cc:686] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [term 1 LEADER]: Becoming Leader. State: Replica: efd7603353df412195c2934295062879, State: Running, Role: LEADER
I20260621 02:49:13.154088 29467 raft_consensus.cc:2454] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efd7603353df412195c2934295062879 in term 1.
I20260621 02:49:13.154779 29671 consensus_queue.cc:237] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601593c14778426fb4ca72255886cb23" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 34113 } } peers { permanent_uuid: "3f2b622a4996405c92db15ce1ea58685" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 46775 } } peers { permanent_uuid: "efd7603353df412195c2934295062879" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 44701 } }
I20260621 02:49:13.159687 29187 catalog_manager.cc:5495] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 reported cstate change: term changed from 0 to 1, leader changed from <none> to efd7603353df412195c2934295062879 (127.9.166.3). New cstate: current_term: 1 leader_uuid: "efd7603353df412195c2934295062879" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601593c14778426fb4ca72255886cb23" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 34113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f2b622a4996405c92db15ce1ea58685" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 46775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efd7603353df412195c2934295062879" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 44701 } health_report { overall_health: HEALTHY } } }
I20260621 02:49:13.233170 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:13.227245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.9.166.3:44701)
0621 02:49:13.227519 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:13.227527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:13.227800 (+ 273us) server_negotiation.cc:407] Connection header received
0621 02:49:13.228055 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:13.228059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:13.228147 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:13.228311 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:49:13.228784 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:13.229517 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:13.230936 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:13.231340 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:13.231532 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:13.232005 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:49:13.232540 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:49:13.232704 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:13.232807 (+ 103us) server_negotiation.cc:299] Negotiation successful
0621 02:49:13.232893 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260621 02:49:13.241156 29333 raft_consensus.cc:1261] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23 [term 1 FOLLOWER]: Refusing update from remote peer efd7603353df412195c2934295062879: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:49:13.241205 29467 raft_consensus.cc:1261] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685 [term 1 FOLLOWER]: Refusing update from remote peer efd7603353df412195c2934295062879: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:49:13.242147 29674 consensus_queue.cc:1035] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "601593c14778426fb4ca72255886cb23" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 34113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:49:13.242466 29671 consensus_queue.cc:1035] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f2b622a4996405c92db15ce1ea58685" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:49:13.256470 29678 mvcc.cc:205] Tried to move back new op lower bound from 7299113587666288640 to 7299113587326320640. Current Snapshot: MvccSnapshot[applied={T|T < 7299113587666288640 or (T in {7299113587666288640})}]
I20260621 02:49:13.258458 29680 mvcc.cc:205] Tried to move back new op lower bound from 7299113587666288640 to 7299113587326320640. Current Snapshot: MvccSnapshot[applied={T|T < 7299113587666288640}]
I20260621 02:49:13.270668 29188 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:49:13.270943 29188 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:49:13.274920 29188 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c9f534fad6f490e8c184dad007c75bf: Sending DeleteTablet for 3 replicas of tablet 2ef267b1884142599b46a05befcbc615
I20260621 02:49:13.276209 29313 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef267b1884142599b46a05befcbc615 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:49:13 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:38848
I20260621 02:49:13.276237 29447 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef267b1884142599b46a05befcbc615 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:49:13 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:42426
I20260621 02:49:13.276432 29580 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef267b1884142599b46a05befcbc615 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:49:13 UTC) from {username='kudu', principal='kudu/127.9.166.62@KRBTEST.COM'} at 127.0.0.1:33792
I20260621 02:49:13.279078 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 29250
I20260621 02:49:13.285284 29690 tablet_replica.cc:330] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879: stopping tablet replica
I20260621 02:49:13.285817 29690 raft_consensus.cc:2229] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:49:13.286727 29690 raft_consensus.cc:2258] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:49:13.289523 29690 ts_tablet_manager.cc:1905] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:49:13.291872 29688 tablet_replica.cc:330] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23: stopping tablet replica
I20260621 02:49:13.292305 29688 raft_consensus.cc:2229] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:49:13.292799 29688 raft_consensus.cc:2258] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:49:13.294862 29688 ts_tablet_manager.cc:1905] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:49:13.299690 29690 ts_tablet_manager.cc:1918] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:49:13.300336 29689 tablet_replica.cc:330] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685: stopping tablet replica
I20260621 02:49:13.300689 29689 raft_consensus.cc:2229] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:49:13.301126 29689 raft_consensus.cc:2258] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:49:13.301512 29690 log.cc:1196] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/2ef267b1884142599b46a05befcbc615
I20260621 02:49:13.301645 29688 ts_tablet_manager.cc:1918] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:49:13.301801 29688 log.cc:1196] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-0/wal/wals/2ef267b1884142599b46a05befcbc615
I20260621 02:49:13.302075 29690 ts_tablet_manager.cc:1939] T 2ef267b1884142599b46a05befcbc615 P efd7603353df412195c2934295062879: Deleting consensus metadata
I20260621 02:49:13.302322 29688 ts_tablet_manager.cc:1939] T 2ef267b1884142599b46a05befcbc615 P 601593c14778426fb4ca72255886cb23: Deleting consensus metadata
I20260621 02:49:13.303642 29174 catalog_manager.cc:4841] TS efd7603353df412195c2934295062879 (127.9.166.3:44701): tablet 2ef267b1884142599b46a05befcbc615 (table test-table [id=2660ae142a4c46dd806c6956acdace57]) successfully deleted
I20260621 02:49:13.304471 29176 catalog_manager.cc:4841] TS 601593c14778426fb4ca72255886cb23 (127.9.166.1:34113): tablet 2ef267b1884142599b46a05befcbc615 (table test-table [id=2660ae142a4c46dd806c6956acdace57]) successfully deleted
I20260621 02:49:13.305866 29689 ts_tablet_manager.cc:1905] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:49:13.311013 29689 ts_tablet_manager.cc:1918] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:49:13.311158 29689 log.cc:1196] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/2ef267b1884142599b46a05befcbc615
Jun 21 02:49:13 dist-test-slave-hvgn krb5kdc[29144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.1@KRBTEST.COM
I20260621 02:49:13.311690 29689 ts_tablet_manager.cc:1939] T 2ef267b1884142599b46a05befcbc615 P 3f2b622a4996405c92db15ce1ea58685: Deleting consensus metadata
I20260621 02:49:13.313288 29174 catalog_manager.cc:4841] TS 3f2b622a4996405c92db15ce1ea58685 (127.9.166.2:46775): tablet 2ef267b1884142599b46a05befcbc615 (table test-table [id=2660ae142a4c46dd806c6956acdace57]) successfully deleted
I20260621 02:49:13.318889 29660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:13.280368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38860 (local address 127.9.166.1:34113)
0621 02:49:13.280591 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:13.280600 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:13.287682 (+ 7082us) server_negotiation.cc:407] Connection header received
0621 02:49:13.288014 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:13.288023 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:13.288126 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:13.288216 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:13.303565 (+ 15349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:13.304575 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:13.306499 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:13.307151 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:13.307297 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:13.312667 (+ 5370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:13.312683 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:13.312701 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:13.312747 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:13.315941 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:13.316769 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:13.316777 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:13.316780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:13.316907 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:13.317304 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:13.317307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:13.317309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:13.317622 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:13.317836 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:13.318225 (+ 389us) server_negotiation.cc:299] Negotiation successful
0621 02:49:13.318473 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:49:13.319573 29373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:13.407495 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 29250
I20260621 02:49:13.431495 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 29384
Jun 21 02:49:13 dist-test-slave-hvgn krb5kdc[29144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.2@KRBTEST.COM
I20260621 02:49:13.448171 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:13.432279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.9.166.2:46775)
0621 02:49:13.432474 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:13.432479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:13.432504 (+ 25us) server_negotiation.cc:407] Connection header received
0621 02:49:13.432628 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:13.432634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:13.432721 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:13.432831 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:13.434075 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:13.435192 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:13.436605 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:13.437241 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:13.437370 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:13.441399 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:13.441418 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:13.441439 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:13.441486 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:13.445266 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:13.445975 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:13.445985 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:13.445990 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:13.446173 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:13.446507 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:13.446511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:13.446514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:13.446989 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:13.447304 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:13.447525 (+ 221us) server_negotiation.cc:299] Negotiation successful
0621 02:49:13.447752 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:49:13.448904 29507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:13.537856 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 29384
I20260621 02:49:13.555816 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 29517
Jun 21 02:49:13 dist-test-slave-hvgn krb5kdc[29144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.3@KRBTEST.COM
I20260621 02:49:13.573558 29662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:13.556746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33812 (local address 127.9.166.3:44701)
0621 02:49:13.556986 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:13.556992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:13.557015 (+ 23us) server_negotiation.cc:407] Connection header received
0621 02:49:13.557084 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:13.557089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:13.557201 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:13.557330 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:13.558565 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:13.559867 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:13.561070 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:13.561753 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:13.561884 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:13.566268 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:13.566281 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:13.566303 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:13.566358 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:13.570526 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:13.571206 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:13.571215 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:13.571219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:13.571442 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:13.571869 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:13.571874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:13.571877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:13.572338 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:13.572653 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:13.572881 (+ 228us) server_negotiation.cc:299] Negotiation successful
0621 02:49:13.573118 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260621 02:49:13.574381 29640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:13.699705 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 29517
I20260621 02:49:13.719671 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 29157
I20260621 02:49:13.720777 29219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:13.830164 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 29157
2026-06-21T02:49:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4594 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-21T02:49:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:13Z Disabled control of system clock
I20260621 02:49:13.905745 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:41211
--webserver_interface=127.9.166.62
--webserver_port=0
--builtin_ntp_servers=127.9.166.20:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:41211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 02:49:14.227313 29699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:14.227648 29699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:14.227751 29699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:14.234804 29699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:14.234911 29699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:14.234948 29699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:14.235000 29699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:14.235044 29699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:14.244120 29699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:41211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:41211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:14.245348 29699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:14.247148 29699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:14.258911 29704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:14.258837 29705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:14.258850 29707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:14.259684 29699 server_base.cc:1034] running on GCE node
I20260621 02:49:14.260365 29699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:14.261703 29699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:14.262928 29699 hybrid_clock.cc:648] HybridClock initialized: now 1782010154262870 us; error 53 us; skew 500 ppm
I20260621 02:49:14.263296 29699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:49:14.265592 29699 webserver.cc:460] Webserver started at http://127.9.166.62:41955/ using document root <none> and password file <none>
I20260621 02:49:14.266283 29699 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:14.266397 29699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:14.266757 29699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:14.269585 29699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "ade2da0212bd4d859ff16d1d9d55dbef"
format_stamp: "Formatted at 2026-06-21 02:49:14 on dist-test-slave-hvgn"
server_key: "6e506f9753484ac2b3cf9883e2143ddb"
server_key_iv: "f2eee5084edb3ddaac1d9b75b1976f13"
server_key_version: "encryptionkey@0"
I20260621 02:49:14.270385 29699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "ade2da0212bd4d859ff16d1d9d55dbef"
format_stamp: "Formatted at 2026-06-21 02:49:14 on dist-test-slave-hvgn"
server_key: "6e506f9753484ac2b3cf9883e2143ddb"
server_key_iv: "f2eee5084edb3ddaac1d9b75b1976f13"
server_key_version: "encryptionkey@0"
I20260621 02:49:14.275368 29699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 02:49:14.278633 29713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:14.280087 29699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 02:49:14.280272 29699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "ade2da0212bd4d859ff16d1d9d55dbef"
format_stamp: "Formatted at 2026-06-21 02:49:14 on dist-test-slave-hvgn"
server_key: "6e506f9753484ac2b3cf9883e2143ddb"
server_key_iv: "f2eee5084edb3ddaac1d9b75b1976f13"
server_key_version: "encryptionkey@0"
I20260621 02:49:14.280478 29699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:14.293869 29699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:14.294585 29699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:14.294855 29699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:14.311942 29699 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:41211
I20260621 02:49:14.311973 29764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:41211 every 8 connection(s)
I20260621 02:49:14.313225 29699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:14.316725 29765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:14.317554 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 29699
I20260621 02:49:14.317737 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:14.318289 9880 external_mini_cluster.cc:1428] Setting key 447a45bd796260e899e5b2a9c83e17f1
I20260621 02:49:14.328769 29765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef: Bootstrap starting.
I20260621 02:49:14.331120 29768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:14.320190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.9.166.62:41211)
0621 02:49:14.321159 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:14.321170 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:14.321227 (+ 57us) server_negotiation.cc:407] Connection header received
0621 02:49:14.322981 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:14.323003 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:14.323523 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:14.323912 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:14.324442 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:14.325648 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:14.326988 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:14.327692 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:14.327881 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:14.329020 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:14.329028 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:14.329040 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:14.329541 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:14.329611 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:14.329631 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:14.329765 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:14.329975 (+ 210us) server_negotiation.cc:299] Negotiation successful
0621 02:49:14.330267 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":359,"threads_started":1}
I20260621 02:49:14.332006 29765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:14.334215 29765 log.cc:824] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:14.337095 29765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef: No bootstrap required, opened a new log
I20260621 02:49:14.344897 29765 raft_consensus.cc:348] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade2da0212bd4d859ff16d1d9d55dbef" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41211 } }
I20260621 02:49:14.345213 29765 raft_consensus.cc:374] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:14.345291 29765 raft_consensus.cc:729] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ade2da0212bd4d859ff16d1d9d55dbef, State: Initialized, Role: FOLLOWER
I20260621 02:49:14.345736 29765 consensus_queue.cc:260] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade2da0212bd4d859ff16d1d9d55dbef" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41211 } }
I20260621 02:49:14.345927 29765 raft_consensus.cc:388] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:14.346016 29765 raft_consensus.cc:482] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:14.346131 29765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:14.348215 29765 raft_consensus.cc:504] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade2da0212bd4d859ff16d1d9d55dbef" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41211 } }
I20260621 02:49:14.348659 29765 leader_election.cc:302] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ade2da0212bd4d859ff16d1d9d55dbef; no voters:
I20260621 02:49:14.349212 29765 leader_election.cc:288] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:14.349385 29770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:14.350607 29770 raft_consensus.cc:686] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef [term 1 LEADER]: Becoming Leader. State: Replica: ade2da0212bd4d859ff16d1d9d55dbef, State: Running, Role: LEADER
I20260621 02:49:14.351090 29770 consensus_queue.cc:237] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade2da0212bd4d859ff16d1d9d55dbef" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41211 } }
I20260621 02:49:14.351562 29765 sys_catalog.cc:564] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:14.353977 29772 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef [sys.catalog]: SysCatalogTable state changed. Reason: New leader ade2da0212bd4d859ff16d1d9d55dbef. Latest consensus state: current_term: 1 leader_uuid: "ade2da0212bd4d859ff16d1d9d55dbef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade2da0212bd4d859ff16d1d9d55dbef" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41211 } } }
I20260621 02:49:14.354336 29772 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:14.354785 29771 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ade2da0212bd4d859ff16d1d9d55dbef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade2da0212bd4d859ff16d1d9d55dbef" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41211 } } }
I20260621 02:49:14.355098 29771 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:14.355300 29777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:14.361789 29777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:14.371788 29777 catalog_manager.cc:1347] Generated new cluster ID: 728375d03e1b4674b7e40ecd28b146b9
I20260621 02:49:14.371892 29777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:14.409253 29777 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:14.411127 29777 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:14.421800 29777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef: Generated new TSK 0
I20260621 02:49:14.422793 29777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:14.435927 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.1:0
--local_ip_for_outbound_sockets=127.9.166.1
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:41211
--builtin_ntp_servers=127.9.166.20:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260621 02:49:14.780066 29789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:14.780339 29789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:14.780489 29789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:14.787853 29789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:14.787942 29789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:14.788143 29789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.1
I20260621 02:49:14.797024 29789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.166.1
--webserver_port=0
--tserver_master_addrs=127.9.166.62:41211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.1
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:14.798295 29789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:14.800065 29789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:14.812845 29794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:14.812815 29795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:14.813378 29797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:14.816833 29789 server_base.cc:1034] running on GCE node
I20260621 02:49:14.817452 29789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:14.818387 29789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:14.819618 29789 hybrid_clock.cc:648] HybridClock initialized: now 1782010154819563 us; error 65 us; skew 500 ppm
I20260621 02:49:14.819968 29789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:49:14.822295 29789 webserver.cc:460] Webserver started at http://127.9.166.1:44689/ using document root <none> and password file <none>
I20260621 02:49:14.822989 29789 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:14.823122 29789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:14.823493 29789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:14.826354 29789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-0/data/instance:
uuid: "de3c08fe28594f4f851d0b6f63b46124"
format_stamp: "Formatted at 2026-06-21 02:49:14 on dist-test-slave-hvgn"
server_key: "52a5cccd0c86d07e60beb65c58ab3ab4"
server_key_iv: "98d7fe00747ac8f1e3bd472acf4256bf"
server_key_version: "encryptionkey@0"
I20260621 02:49:14.827163 29789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance:
uuid: "de3c08fe28594f4f851d0b6f63b46124"
format_stamp: "Formatted at 2026-06-21 02:49:14 on dist-test-slave-hvgn"
server_key: "52a5cccd0c86d07e60beb65c58ab3ab4"
server_key_iv: "98d7fe00747ac8f1e3bd472acf4256bf"
server_key_version: "encryptionkey@0"
I20260621 02:49:14.832420 29789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.002s
I20260621 02:49:14.836050 29803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:14.837459 29789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:49:14.837626 29789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-0/wal
uuid: "de3c08fe28594f4f851d0b6f63b46124"
format_stamp: "Formatted at 2026-06-21 02:49:14 on dist-test-slave-hvgn"
server_key: "52a5cccd0c86d07e60beb65c58ab3ab4"
server_key_iv: "98d7fe00747ac8f1e3bd472acf4256bf"
server_key_version: "encryptionkey@0"
I20260621 02:49:14.837855 29789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:14.869108 29789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:14.869990 29789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:14.870272 29789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:14.871347 29789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:49:14.872862 29789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:49:14.872972 29789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:14.873102 29789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:49:14.873198 29789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:14.906774 29789 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.1:44415
I20260621 02:49:14.906867 29915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.1:44415 every 8 connection(s)
I20260621 02:49:14.908159 29789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-0/data/info.pb
I20260621 02:49:14.913018 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 29789
I20260621 02:49:14.913206 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-0/wal/instance
I20260621 02:49:14.913683 9880 external_mini_cluster.cc:1428] Setting key 788fe6e726acfa544a949c767281109e
I20260621 02:49:14.916661 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.2:0
--local_ip_for_outbound_sockets=127.9.166.2
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:41211
--builtin_ntp_servers=127.9.166.20:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 02:49:14.921911 29918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:14.910800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.1:36905 (local address 127.9.166.62:41211)
0621 02:49:14.911237 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:14.911242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:14.912511 (+ 1269us) server_negotiation.cc:407] Connection header received
0621 02:49:14.914539 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:14.914543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:14.914639 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:14.914761 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:14.915927 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:14.917068 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:14.918617 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:14.919320 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:14.919505 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:14.920853 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:14.920862 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:14.920870 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:14.920898 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:14.920970 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:14.920989 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:14.921079 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:14.921311 (+ 232us) server_negotiation.cc:299] Negotiation successful
0621 02:49:14.921425 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":217,"threads_started":1}
I20260621 02:49:14.922787 29919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:14.911486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:41211 (local address 127.9.166.1:36905)
0621 02:49:14.912361 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:14.912397 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:14.914232 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:14.914960 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:14.914968 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:14.915054 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:14.915758 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:14.915776 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:14.917215 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:14.917219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:14.918475 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:14.918488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:14.919586 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:14.919592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:14.919795 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:14.920514 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:49:14.920516 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:14.920556 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:49:14.920641 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:14.921094 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:14.921109 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:14.921115 (+ 6us) client_negotiation.cc:769] Sending connection context
0621 02:49:14.921306 (+ 191us) client_negotiation.cc:240] Negotiation successful
0621 02:49:14.921697 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":260,"threads_started":1}
I20260621 02:49:14.924310 29916 heartbeater.cc:347] Connected to a master server at 127.9.166.62:41211
I20260621 02:49:14.924645 29916 heartbeater.cc:464] Registering TS with master...
I20260621 02:49:14.925420 29916 heartbeater.cc:511] Master 127.9.166.62:41211 requested a full tablet report, sending...
I20260621 02:49:14.928047 29730 ts_manager.cc:194] Registered new tserver with Master: de3c08fe28594f4f851d0b6f63b46124 (127.9.166.1:44415)
I20260621 02:49:14.929464 29730 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.166.1:36905
W20260621 02:49:15.241297 29920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:15.241653 29920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:15.241814 29920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:15.249138 29920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:15.249265 29920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:15.249446 29920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.2
I20260621 02:49:15.258710 29920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.166.2
--webserver_port=0
--tserver_master_addrs=127.9.166.62:41211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.2
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:15.260026 29920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:15.261781 29920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:15.277648 29928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:15.277742 29925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:15.277685 29926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:15.278399 29920 server_base.cc:1034] running on GCE node
I20260621 02:49:15.279019 29920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:15.279968 29920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:15.281178 29920 hybrid_clock.cc:648] HybridClock initialized: now 1782010155281104 us; error 63 us; skew 500 ppm
I20260621 02:49:15.281450 29920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:49:15.283731 29920 webserver.cc:460] Webserver started at http://127.9.166.2:38995/ using document root <none> and password file <none>
I20260621 02:49:15.284430 29920 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:15.284514 29920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:15.284777 29920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:15.287559 29920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-1/data/instance:
uuid: "f536f23cf6d64367a1f980eb6a908ea5"
format_stamp: "Formatted at 2026-06-21 02:49:15 on dist-test-slave-hvgn"
server_key: "212782e115c10084fd742c19216fa111"
server_key_iv: "5a83e501a02217d09e0f29ba8e27ca24"
server_key_version: "encryptionkey@0"
I20260621 02:49:15.288285 29920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance:
uuid: "f536f23cf6d64367a1f980eb6a908ea5"
format_stamp: "Formatted at 2026-06-21 02:49:15 on dist-test-slave-hvgn"
server_key: "212782e115c10084fd742c19216fa111"
server_key_iv: "5a83e501a02217d09e0f29ba8e27ca24"
server_key_version: "encryptionkey@0"
I20260621 02:49:15.293097 29920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:49:15.296401 29934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:15.297721 29920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:49:15.297879 29920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-1/wal
uuid: "f536f23cf6d64367a1f980eb6a908ea5"
format_stamp: "Formatted at 2026-06-21 02:49:15 on dist-test-slave-hvgn"
server_key: "212782e115c10084fd742c19216fa111"
server_key_iv: "5a83e501a02217d09e0f29ba8e27ca24"
server_key_version: "encryptionkey@0"
I20260621 02:49:15.298084 29920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:15.318444 29920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:15.319200 29920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:15.319525 29920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:15.320559 29920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:49:15.321985 29920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:49:15.322103 29920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:15.322227 29920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:49:15.322319 29920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:15.355684 29920 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.2:36825
I20260621 02:49:15.355786 30046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.2:36825 every 8 connection(s)
I20260621 02:49:15.356986 29920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-1/data/info.pb
I20260621 02:49:15.363461 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 29920
I20260621 02:49:15.363636 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-1/wal/instance
I20260621 02:49:15.364148 9880 external_mini_cluster.cc:1428] Setting key 0b0da8cb3feb2aaed75e06330b458b3b
I20260621 02:49:15.367313 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.166.3:0
--local_ip_for_outbound_sockets=127.9.166.3
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:41211
--builtin_ntp_servers=127.9.166.20:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260621 02:49:15.371191 29918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:15.359788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.2:60181 (local address 127.9.166.62:41211)
0621 02:49:15.359974 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:15.359978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:15.361365 (+ 1387us) server_negotiation.cc:407] Connection header received
0621 02:49:15.363098 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:15.363102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:15.363167 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:15.363261 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:15.364743 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.365762 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.368214 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.368621 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.368733 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:15.370433 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:15.370438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:15.370442 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:15.370459 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:15.370492 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:15.370503 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:15.370575 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:15.370869 (+ 294us) server_negotiation.cc:299] Negotiation successful
0621 02:49:15.370949 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260621 02:49:15.372648 30049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:15.360321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:41211 (local address 127.9.166.2:60181)
0621 02:49:15.361212 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:15.361250 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:15.362835 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:15.363565 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:15.363578 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:15.363721 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:15.364536 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:15.364563 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.365959 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.365966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:15.368043 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:15.368060 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.368885 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.368894 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:15.369223 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:15.370058 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:49:15.370064 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:15.370116 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:49:15.370222 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:15.370603 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:15.370616 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:15.370627 (+ 11us) client_negotiation.cc:769] Sending connection context
0621 02:49:15.370816 (+ 189us) client_negotiation.cc:240] Negotiation successful
0621 02:49:15.371231 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":252,"threads_started":1}
I20260621 02:49:15.374174 30047 heartbeater.cc:347] Connected to a master server at 127.9.166.62:41211
I20260621 02:49:15.374490 30047 heartbeater.cc:464] Registering TS with master...
I20260621 02:49:15.375347 30047 heartbeater.cc:511] Master 127.9.166.62:41211 requested a full tablet report, sending...
I20260621 02:49:15.377008 29730 ts_manager.cc:194] Registered new tserver with Master: f536f23cf6d64367a1f980eb6a908ea5 (127.9.166.2:36825)
I20260621 02:49:15.378072 29730 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.166.2:60181
W20260621 02:49:15.692035 30050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:15.692349 30050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:15.692512 30050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:15.699599 30050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:15.699728 30050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:15.699894 30050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.166.3
I20260621 02:49:15.708937 30050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.166.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.166.3
--webserver_port=0
--tserver_master_addrs=127.9.166.62:41211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.166.3
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:15.710239 30050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:15.712013 30050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:15.724679 30055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:15.724709 30056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:15.724591 30058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:15.725605 30050 server_base.cc:1034] running on GCE node
I20260621 02:49:15.726253 30050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:15.727159 30050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:15.728395 30050 hybrid_clock.cc:648] HybridClock initialized: now 1782010155728334 us; error 57 us; skew 500 ppm
I20260621 02:49:15.728706 30050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260621 02:49:15.731060 30050 webserver.cc:460] Webserver started at http://127.9.166.3:44907/ using document root <none> and password file <none>
I20260621 02:49:15.731806 30050 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:15.731937 30050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:15.732257 30050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:15.734997 30050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-2/data/instance:
uuid: "ec4787c5e62c426cb21ad8f8ee0113b7"
format_stamp: "Formatted at 2026-06-21 02:49:15 on dist-test-slave-hvgn"
server_key: "cb7f58db533aae4a93cdbaef566ec3ad"
server_key_iv: "8c51697dbfe52fdbc1242183543fb0b1"
server_key_version: "encryptionkey@0"
I20260621 02:49:15.735810 30050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance:
uuid: "ec4787c5e62c426cb21ad8f8ee0113b7"
format_stamp: "Formatted at 2026-06-21 02:49:15 on dist-test-slave-hvgn"
server_key: "cb7f58db533aae4a93cdbaef566ec3ad"
server_key_iv: "8c51697dbfe52fdbc1242183543fb0b1"
server_key_version: "encryptionkey@0"
I20260621 02:49:15.740731 30050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260621 02:49:15.744427 30064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:15.745738 30050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260621 02:49:15.745911 30050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-2/wal
uuid: "ec4787c5e62c426cb21ad8f8ee0113b7"
format_stamp: "Formatted at 2026-06-21 02:49:15 on dist-test-slave-hvgn"
server_key: "cb7f58db533aae4a93cdbaef566ec3ad"
server_key_iv: "8c51697dbfe52fdbc1242183543fb0b1"
server_key_version: "encryptionkey@0"
I20260621 02:49:15.746124 30050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:15.766316 30050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:15.767133 30050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:15.767442 30050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:15.768404 30050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260621 02:49:15.769814 30050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260621 02:49:15.769933 30050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:15.770068 30050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260621 02:49:15.770159 30050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:15.803185 30050 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.3:38519
I20260621 02:49:15.803314 30176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.3:38519 every 8 connection(s)
I20260621 02:49:15.804507 30050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-2/data/info.pb
I20260621 02:49:15.813691 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 30050
I20260621 02:49:15.813884 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-2/wal/instance
I20260621 02:49:15.814432 9880 external_mini_cluster.cc:1428] Setting key e15572f179108460b9e790c57c44e987
I20260621 02:49:15.817356 29918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:15.806967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:48589 (local address 127.9.166.62:41211)
0621 02:49:15.807137 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:15.807142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:15.808509 (+ 1367us) server_negotiation.cc:407] Connection header received
0621 02:49:15.810413 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:15.810421 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:15.810512 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:15.810622 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:15.811827 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.812557 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.814124 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.814776 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.814920 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:15.816543 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:15.816548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:15.816551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:15.816567 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:15.816596 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:15.816608 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:15.816677 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:15.816950 (+ 273us) server_negotiation.cc:299] Negotiation successful
0621 02:49:15.817027 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 02:49:15.818310 30179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:15.807491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:41211 (local address 127.9.166.3:48589)
0621 02:49:15.808351 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:15.808387 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:15.810149 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:15.810855 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:15.810863 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:15.810948 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:15.811630 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:15.811671 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.812689 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.812692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:15.813979 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:15.813995 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.815025 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.815032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:15.815286 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:15.816226 (+ 940us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0621 02:49:15.816233 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:15.816299 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0621 02:49:15.816380 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:15.816702 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:15.816712 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:15.816745 (+ 33us) client_negotiation.cc:769] Sending connection context
0621 02:49:15.816920 (+ 175us) client_negotiation.cc:240] Negotiation successful
0621 02:49:15.817317 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":254,"threads_started":1}
I20260621 02:49:15.819561 30177 heartbeater.cc:347] Connected to a master server at 127.9.166.62:41211
I20260621 02:49:15.819886 30177 heartbeater.cc:464] Registering TS with master...
I20260621 02:49:15.820600 30177 heartbeater.cc:511] Master 127.9.166.62:41211 requested a full tablet report, sending...
I20260621 02:49:15.821873 29730 ts_manager.cc:194] Registered new tserver with Master: ec4787c5e62c426cb21ad8f8ee0113b7 (127.9.166.3:38519)
I20260621 02:49:15.822757 29730 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.166.3:48589
I20260621 02:49:15.829550 9880 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260621 02:49:15.839184 29918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:15.833651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44956 (local address 127.9.166.62:41211)
0621 02:49:15.833854 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:15.833858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:15.834371 (+ 513us) server_negotiation.cc:407] Connection header received
0621 02:49:15.834488 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:15.834491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:15.834557 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:15.834635 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:15.835154 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.835896 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.837177 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.837568 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.837654 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:15.838622 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:15.838626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:15.838629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:15.838646 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:15.838674 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:15.838686 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:15.838773 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:15.838908 (+ 135us) server_negotiation.cc:299] Negotiation successful
0621 02:49:15.838977 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:49:15.843070 29730 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260621 02:49:15.850132 29730 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260621 02:49:15.880267 30189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:15.861906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:44415 (local address 127.0.0.1:59698)
0621 02:49:15.873412 (+ 11506us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:15.873459 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:15.873633 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:15.874276 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:15.874283 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:15.874314 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:15.874924 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:15.874938 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.876486 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.876493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:15.877977 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:15.877989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.879298 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.879305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:15.879533 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:15.879594 (+ 61us) client_negotiation.cc:769] Sending connection context
0621 02:49:15.879710 (+ 116us) client_negotiation.cc:240] Negotiation successful
0621 02:49:15.879915 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11267,"thread_start_us":11110,"threads_started":1}
I20260621 02:49:15.881218 30190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:15.862064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59698 (local address 127.9.166.1:44415)
0621 02:49:15.872921 (+ 10857us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:15.872934 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:15.873725 (+ 791us) server_negotiation.cc:407] Connection header received
0621 02:49:15.873858 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:15.873866 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:15.874073 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:15.874290 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:15.875138 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.876357 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.878157 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.879099 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.879248 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:15.880582 (+ 1334us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:15.880667 (+ 85us) server_negotiation.cc:299] Negotiation successful
0621 02:49:15.880868 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10701,"thread_start_us":10575,"threads_started":1}
I20260621 02:49:15.884701 29851 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe04d5836c745ed9d932f9abbd4574a (DEFAULT_TABLE table=test-table [id=6370a4f548ff4fb7bb72e32690e711bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:49:15.886204 29851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe04d5836c745ed9d932f9abbd4574a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:15.895126 30192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:15.862346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:36825 (local address 127.0.0.1:38492)
0621 02:49:15.887037 (+ 24691us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:15.887064 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:15.887220 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:15.888099 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:15.888106 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:15.888136 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:15.888506 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:15.888518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.890398 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.890403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:15.892293 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:15.892305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.894367 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.894373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:15.894543 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:15.894577 (+ 34us) client_negotiation.cc:769] Sending connection context
0621 02:49:15.894675 (+ 98us) client_negotiation.cc:240] Negotiation successful
0621 02:49:15.894816 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24526,"thread_start_us":26156,"threads_started":1}
I20260621 02:49:15.895234 30194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:15.862925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38492 (local address 127.9.166.2:36825)
0621 02:49:15.886364 (+ 23439us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:15.886375 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:15.887199 (+ 824us) server_negotiation.cc:407] Connection header received
0621 02:49:15.887423 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:15.887432 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:15.887679 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:15.887881 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:15.888714 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.890249 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.892547 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.893848 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.893996 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:15.894244 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:15.894755 (+ 511us) server_negotiation.cc:299] Negotiation successful
0621 02:49:15.894926 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23225,"thread_start_us":23448,"threads_started":1}
I20260621 02:49:15.896037 30189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:15.862084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.3:38519 (local address 127.0.0.1:48368)
0621 02:49:15.881019 (+ 18935us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:15.881036 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:15.881128 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:15.888801 (+ 7673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:15.888809 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:15.888846 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:15.889122 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:15.889137 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.891845 (+ 2708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.891852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:15.893806 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:15.893820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.895577 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.895582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:15.895707 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:15.895730 (+ 23us) client_negotiation.cc:769] Sending connection context
0621 02:49:15.895778 (+ 48us) client_negotiation.cc:240] Negotiation successful
0621 02:49:15.895865 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18818,"thread_start_us":27869,"threads_started":1}
I20260621 02:49:15.898370 29982 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe04d5836c745ed9d932f9abbd4574a (DEFAULT_TABLE table=test-table [id=6370a4f548ff4fb7bb72e32690e711bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:49:15.899623 29982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe04d5836c745ed9d932f9abbd4574a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:15.900130 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:15.862645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.9.166.3:38519)
0621 02:49:15.887832 (+ 25187us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:15.887843 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:15.887884 (+ 41us) server_negotiation.cc:407] Connection header received
0621 02:49:15.887999 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:15.888010 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:15.888312 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:15.889344 (+ 1032us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:15.889685 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.891692 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.893975 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:15.895405 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:15.899080 (+ 3675us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:15.899465 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:15.899552 (+ 87us) server_negotiation.cc:299] Negotiation successful
0621 02:49:15.899783 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25001,"thread_start_us":24860,"threads_started":1}
I20260621 02:49:15.900416 30195 tablet_bootstrap.cc:492] T 4fe04d5836c745ed9d932f9abbd4574a P de3c08fe28594f4f851d0b6f63b46124: Bootstrap starting.
I20260621 02:49:15.903805 30195 tablet_bootstrap.cc:654] T 4fe04d5836c745ed9d932f9abbd4574a P de3c08fe28594f4f851d0b6f63b46124: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:15.903929 30112 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe04d5836c745ed9d932f9abbd4574a (DEFAULT_TABLE table=test-table [id=6370a4f548ff4fb7bb72e32690e711bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260621 02:49:15.904838 30195 log.cc:824] T 4fe04d5836c745ed9d932f9abbd4574a P de3c08fe28594f4f851d0b6f63b46124: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:15.905272 30112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe04d5836c745ed9d932f9abbd4574a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:15.907913 30195 tablet_bootstrap.cc:492] T 4fe04d5836c745ed9d932f9abbd4574a P de3c08fe28594f4f851d0b6f63b46124: No bootstrap required, opened a new log
I20260621 02:49:15.908289 30195 ts_tablet_manager.cc:1397] T 4fe04d5836c745ed9d932f9abbd4574a P de3c08fe28594f4f851d0b6f63b46124: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260621 02:49:15.914978 30197 tablet_bootstrap.cc:492] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5: Bootstrap starting.
I20260621 02:49:15.918455 30198 tablet_bootstrap.cc:492] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7: Bootstrap starting.
I20260621 02:49:15.918567 30197 tablet_bootstrap.cc:654] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:15.919826 30197 log.cc:824] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:15.920706 30195 raft_consensus.cc:348] T 4fe04d5836c745ed9d932f9abbd4574a P de3c08fe28594f4f851d0b6f63b46124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3c08fe28594f4f851d0b6f63b46124" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 44415 } } peers { permanent_uuid: "ec4787c5e62c426cb21ad8f8ee0113b7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 38519 } } peers { permanent_uuid: "f536f23cf6d64367a1f980eb6a908ea5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36825 } }
I20260621 02:49:15.921201 30195 raft_consensus.cc:374] T 4fe04d5836c745ed9d932f9abbd4574a P de3c08fe28594f4f851d0b6f63b46124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:15.921348 30195 raft_consensus.cc:729] T 4fe04d5836c745ed9d932f9abbd4574a P de3c08fe28594f4f851d0b6f63b46124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de3c08fe28594f4f851d0b6f63b46124, State: Initialized, Role: FOLLOWER
I20260621 02:49:15.921989 30198 tablet_bootstrap.cc:654] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:15.922019 30195 consensus_queue.cc:260] T 4fe04d5836c745ed9d932f9abbd4574a P de3c08fe28594f4f851d0b6f63b46124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3c08fe28594f4f851d0b6f63b46124" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 44415 } } peers { permanent_uuid: "ec4787c5e62c426cb21ad8f8ee0113b7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 38519 } } peers { permanent_uuid: "f536f23cf6d64367a1f980eb6a908ea5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36825 } }
I20260621 02:49:15.923451 30198 log.cc:824] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:15.929246 29916 heartbeater.cc:503] Master 127.9.166.62:41211 was elected leader, sending a full tablet report...
I20260621 02:49:15.929957 30195 ts_tablet_manager.cc:1428] T 4fe04d5836c745ed9d932f9abbd4574a P de3c08fe28594f4f851d0b6f63b46124: Time spent starting tablet: real 0.022s user 0.013s sys 0.005s
I20260621 02:49:15.930857 30197 tablet_bootstrap.cc:492] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5: No bootstrap required, opened a new log
I20260621 02:49:15.931151 30197 ts_tablet_manager.cc:1397] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260621 02:49:15.931653 30198 tablet_bootstrap.cc:492] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7: No bootstrap required, opened a new log
I20260621 02:49:15.931913 30198 ts_tablet_manager.cc:1397] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260621 02:49:15.938788 30197 raft_consensus.cc:348] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3c08fe28594f4f851d0b6f63b46124" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 44415 } } peers { permanent_uuid: "ec4787c5e62c426cb21ad8f8ee0113b7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 38519 } } peers { permanent_uuid: "f536f23cf6d64367a1f980eb6a908ea5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36825 } }
I20260621 02:49:15.939146 30197 raft_consensus.cc:374] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:15.939261 30197 raft_consensus.cc:729] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f536f23cf6d64367a1f980eb6a908ea5, State: Initialized, Role: FOLLOWER
I20260621 02:49:15.939502 30198 raft_consensus.cc:348] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3c08fe28594f4f851d0b6f63b46124" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 44415 } } peers { permanent_uuid: "ec4787c5e62c426cb21ad8f8ee0113b7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 38519 } } peers { permanent_uuid: "f536f23cf6d64367a1f980eb6a908ea5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36825 } }
I20260621 02:49:15.939813 30198 raft_consensus.cc:374] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:15.939920 30198 raft_consensus.cc:729] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec4787c5e62c426cb21ad8f8ee0113b7, State: Initialized, Role: FOLLOWER
I20260621 02:49:15.939952 30197 consensus_queue.cc:260] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3c08fe28594f4f851d0b6f63b46124" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 44415 } } peers { permanent_uuid: "ec4787c5e62c426cb21ad8f8ee0113b7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 38519 } } peers { permanent_uuid: "f536f23cf6d64367a1f980eb6a908ea5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36825 } }
I20260621 02:49:15.940479 30198 consensus_queue.cc:260] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3c08fe28594f4f851d0b6f63b46124" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 44415 } } peers { permanent_uuid: "ec4787c5e62c426cb21ad8f8ee0113b7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 38519 } } peers { permanent_uuid: "f536f23cf6d64367a1f980eb6a908ea5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36825 } }
I20260621 02:49:15.941576 30177 heartbeater.cc:503] Master 127.9.166.62:41211 was elected leader, sending a full tablet report...
I20260621 02:49:15.941802 30047 heartbeater.cc:503] Master 127.9.166.62:41211 was elected leader, sending a full tablet report...
I20260621 02:49:15.942361 30198 ts_tablet_manager.cc:1428] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20260621 02:49:15.943097 30197 ts_tablet_manager.cc:1428] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
W20260621 02:49:16.056147 30178 tablet.cc:2369] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:49:16.109385 30048 tablet.cc:2369] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260621 02:49:16.160583 29917 tablet.cc:2369] T 4fe04d5836c745ed9d932f9abbd4574a P de3c08fe28594f4f851d0b6f63b46124: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:49:16.160661 30203 raft_consensus.cc:482] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260621 02:49:16.161026 30203 raft_consensus.cc:504] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3c08fe28594f4f851d0b6f63b46124" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 44415 } } peers { permanent_uuid: "ec4787c5e62c426cb21ad8f8ee0113b7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 38519 } } peers { permanent_uuid: "f536f23cf6d64367a1f980eb6a908ea5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36825 } }
I20260621 02:49:16.162660 30203 leader_election.cc:288] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de3c08fe28594f4f851d0b6f63b46124 (127.9.166.1:44415), f536f23cf6d64367a1f980eb6a908ea5 (127.9.166.2:36825)
I20260621 02:49:16.169394 30194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:16.163078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:49623 (local address 127.9.166.2:36825)
0621 02:49:16.163286 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:16.163293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:16.163531 (+ 238us) server_negotiation.cc:407] Connection header received
0621 02:49:16.163794 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:16.163802 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:16.163908 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:16.164028 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:16.164470 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:16.165452 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:16.167528 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:16.168594 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:16.168684 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:16.168748 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:16.169102 (+ 354us) server_negotiation.cc:299] Negotiation successful
0621 02:49:16.169180 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 02:49:16.169557 30179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:16.163098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.2:36825 (local address 127.9.166.3:49623)
0621 02:49:16.163430 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:16.163450 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:16.163596 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:16.164098 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:16.164104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:16.164137 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:16.164306 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:16.164314 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:16.165617 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:16.165623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:16.167318 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:16.167330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:16.168743 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:16.168751 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:16.168942 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:16.168980 (+ 38us) client_negotiation.cc:769] Sending connection context
0621 02:49:16.169076 (+ 96us) client_negotiation.cc:240] Negotiation successful
0621 02:49:16.169230 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":1}
I20260621 02:49:16.170267 30190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:16.163216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.166.3:34947 (local address 127.9.166.1:44415)
0621 02:49:16.163444 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:16.163450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:16.164287 (+ 837us) server_negotiation.cc:407] Connection header received
0621 02:49:16.164526 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:16.164530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:16.164597 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:16.164677 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0621 02:49:16.165293 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:16.166309 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:16.168331 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:16.169377 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:16.169493 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:16.169583 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:16.169915 (+ 332us) server_negotiation.cc:299] Negotiation successful
0621 02:49:16.170035 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:49:16.170290 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:16.163455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:44415 (local address 127.9.166.3:34947)
0621 02:49:16.164174 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:16.164193 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:16.164347 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:16.164751 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:16.164756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:16.164781 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0621 02:49:16.165105 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:16.165116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:16.166466 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:16.166470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:16.168141 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:16.168154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:16.169564 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:16.169571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:16.169771 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:16.169801 (+ 30us) client_negotiation.cc:769] Sending connection context
0621 02:49:16.169890 (+ 89us) client_negotiation.cc:240] Negotiation successful
0621 02:49:16.170006 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":402,"threads_started":1}
I20260621 02:49:16.170801 30002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe04d5836c745ed9d932f9abbd4574a" candidate_uuid: "ec4787c5e62c426cb21ad8f8ee0113b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f536f23cf6d64367a1f980eb6a908ea5" is_pre_election: true
I20260621 02:49:16.171446 30002 raft_consensus.cc:2454] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec4787c5e62c426cb21ad8f8ee0113b7 in term 0.
I20260621 02:49:16.171496 29871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe04d5836c745ed9d932f9abbd4574a" candidate_uuid: "ec4787c5e62c426cb21ad8f8ee0113b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de3c08fe28594f4f851d0b6f63b46124" is_pre_election: true
I20260621 02:49:16.171988 29871 raft_consensus.cc:2454] T 4fe04d5836c745ed9d932f9abbd4574a P de3c08fe28594f4f851d0b6f63b46124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec4787c5e62c426cb21ad8f8ee0113b7 in term 0.
I20260621 02:49:16.172197 30068 leader_election.cc:302] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec4787c5e62c426cb21ad8f8ee0113b7, f536f23cf6d64367a1f980eb6a908ea5; no voters:
I20260621 02:49:16.172611 30203 raft_consensus.cc:2781] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260621 02:49:16.172793 30203 raft_consensus.cc:482] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260621 02:49:16.172899 30203 raft_consensus.cc:3037] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:16.174962 30203 raft_consensus.cc:504] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3c08fe28594f4f851d0b6f63b46124" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 44415 } } peers { permanent_uuid: "ec4787c5e62c426cb21ad8f8ee0113b7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 38519 } } peers { permanent_uuid: "f536f23cf6d64367a1f980eb6a908ea5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36825 } }
I20260621 02:49:16.175592 30203 leader_election.cc:288] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [CANDIDATE]: Term 1 election: Requested vote from peers de3c08fe28594f4f851d0b6f63b46124 (127.9.166.1:44415), f536f23cf6d64367a1f980eb6a908ea5 (127.9.166.2:36825)
I20260621 02:49:16.175916 29871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe04d5836c745ed9d932f9abbd4574a" candidate_uuid: "ec4787c5e62c426cb21ad8f8ee0113b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de3c08fe28594f4f851d0b6f63b46124"
I20260621 02:49:16.176131 29871 raft_consensus.cc:3037] T 4fe04d5836c745ed9d932f9abbd4574a P de3c08fe28594f4f851d0b6f63b46124 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:16.176128 30002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe04d5836c745ed9d932f9abbd4574a" candidate_uuid: "ec4787c5e62c426cb21ad8f8ee0113b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f536f23cf6d64367a1f980eb6a908ea5"
I20260621 02:49:16.176338 30002 raft_consensus.cc:3037] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:16.178344 29871 raft_consensus.cc:2454] T 4fe04d5836c745ed9d932f9abbd4574a P de3c08fe28594f4f851d0b6f63b46124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec4787c5e62c426cb21ad8f8ee0113b7 in term 1.
I20260621 02:49:16.178517 30002 raft_consensus.cc:2454] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec4787c5e62c426cb21ad8f8ee0113b7 in term 1.
I20260621 02:49:16.178733 30067 leader_election.cc:302] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de3c08fe28594f4f851d0b6f63b46124, ec4787c5e62c426cb21ad8f8ee0113b7; no voters:
I20260621 02:49:16.179028 30203 raft_consensus.cc:2781] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:16.179519 30203 raft_consensus.cc:686] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [term 1 LEADER]: Becoming Leader. State: Replica: ec4787c5e62c426cb21ad8f8ee0113b7, State: Running, Role: LEADER
I20260621 02:49:16.180049 30203 consensus_queue.cc:237] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3c08fe28594f4f851d0b6f63b46124" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 44415 } } peers { permanent_uuid: "ec4787c5e62c426cb21ad8f8ee0113b7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 38519 } } peers { permanent_uuid: "f536f23cf6d64367a1f980eb6a908ea5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36825 } }
I20260621 02:49:16.184489 29730 catalog_manager.cc:5495] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec4787c5e62c426cb21ad8f8ee0113b7 (127.9.166.3). New cstate: current_term: 1 leader_uuid: "ec4787c5e62c426cb21ad8f8ee0113b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3c08fe28594f4f851d0b6f63b46124" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 44415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec4787c5e62c426cb21ad8f8ee0113b7" member_type: VOTER last_known_addr { host: "127.9.166.3" port: 38519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f536f23cf6d64367a1f980eb6a908ea5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36825 } health_report { overall_health: UNKNOWN } } }
I20260621 02:49:16.238632 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:16.233515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.9.166.3:38519)
0621 02:49:16.233688 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:16.233693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:16.233716 (+ 23us) server_negotiation.cc:407] Connection header received
0621 02:49:16.233845 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:16.233849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:16.233934 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:16.234059 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:49:16.234399 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:16.235146 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:16.236574 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:16.236974 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:16.237066 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:16.237615 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:49:16.238147 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:49:16.238252 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:16.238382 (+ 130us) server_negotiation.cc:299] Negotiation successful
0621 02:49:16.238444 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260621 02:49:16.246778 29871 raft_consensus.cc:1261] T 4fe04d5836c745ed9d932f9abbd4574a P de3c08fe28594f4f851d0b6f63b46124 [term 1 FOLLOWER]: Refusing update from remote peer ec4787c5e62c426cb21ad8f8ee0113b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:49:16.246827 30002 raft_consensus.cc:1261] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5 [term 1 FOLLOWER]: Refusing update from remote peer ec4787c5e62c426cb21ad8f8ee0113b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:49:16.247838 30205 consensus_queue.cc:1035] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de3c08fe28594f4f851d0b6f63b46124" member_type: VOTER last_known_addr { host: "127.9.166.1" port: 44415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:49:16.248221 30203 consensus_queue.cc:1035] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f536f23cf6d64367a1f980eb6a908ea5" member_type: VOTER last_known_addr { host: "127.9.166.2" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:49:16.260352 30209 mvcc.cc:205] Tried to move back new op lower bound from 7299113599977148416 to 7299113599717351424. Current Snapshot: MvccSnapshot[applied={T|T < 7299113599977148416}]
I20260621 02:49:16.260713 30208 mvcc.cc:205] Tried to move back new op lower bound from 7299113599977148416 to 7299113599717351424. Current Snapshot: MvccSnapshot[applied={T|T < 7299113599977148416}]
I20260621 02:49:16.272388 29730 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:49:16.272662 29730 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260621 02:49:16.276223 29730 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ade2da0212bd4d859ff16d1d9d55dbef: Sending DeleteTablet for 3 replicas of tablet 4fe04d5836c745ed9d932f9abbd4574a
I20260621 02:49:16.277351 30112 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe04d5836c745ed9d932f9abbd4574a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:49:16 UTC) from {username='slave'} at 127.0.0.1:48368
I20260621 02:49:16.277345 29851 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe04d5836c745ed9d932f9abbd4574a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:49:16 UTC) from {username='slave'} at 127.0.0.1:59698
I20260621 02:49:16.278362 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 29789
I20260621 02:49:16.281824 29982 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe04d5836c745ed9d932f9abbd4574a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-21 02:49:16 UTC) from {username='slave'} at 127.0.0.1:38492
I20260621 02:49:16.305279 30221 tablet_replica.cc:330] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5: stopping tablet replica
I20260621 02:49:16.307132 30221 raft_consensus.cc:2229] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:49:16.310992 30221 raft_consensus.cc:2258] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:49:16.313972 30219 tablet_replica.cc:330] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7: stopping tablet replica
I20260621 02:49:16.314540 30219 raft_consensus.cc:2229] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [term 1 LEADER]: Raft consensus shutting down.
I20260621 02:49:16.315289 30219 raft_consensus.cc:2258] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:49:16.318553 30219 ts_tablet_manager.cc:1905] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:49:16.319211 30221 ts_tablet_manager.cc:1905] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260621 02:49:16.324146 30190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:16.279710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59710 (local address 127.9.166.1:44415)
0621 02:49:16.279895 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:16.279901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:16.304550 (+ 24649us) server_negotiation.cc:407] Connection header received
0621 02:49:16.305752 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:16.305759 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:16.305906 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:16.305994 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:16.307426 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:16.308658 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:16.311792 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:16.312463 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:16.312562 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:16.317411 (+ 4849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:16.317423 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:16.317440 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:16.318262 (+ 822us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:16.318324 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:16.318349 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:16.318459 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:16.323621 (+ 5162us) server_negotiation.cc:299] Negotiation successful
0621 02:49:16.323755 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:49:16.325534 29911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:16.326001 30219 ts_tablet_manager.cc:1918] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:49:16.326164 30219 log.cc:1196] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-2/wal/wals/4fe04d5836c745ed9d932f9abbd4574a
I20260621 02:49:16.326678 30219 ts_tablet_manager.cc:1939] T 4fe04d5836c745ed9d932f9abbd4574a P ec4787c5e62c426cb21ad8f8ee0113b7: Deleting consensus metadata
I20260621 02:49:16.328459 29714 catalog_manager.cc:4841] TS ec4787c5e62c426cb21ad8f8ee0113b7 (127.9.166.3:38519): tablet 4fe04d5836c745ed9d932f9abbd4574a (table test-table [id=6370a4f548ff4fb7bb72e32690e711bd]) successfully deleted
I20260621 02:49:16.331324 30221 ts_tablet_manager.cc:1918] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260621 02:49:16.331524 30221 log.cc:1196] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5: Deleting WAL directory at /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1782009844354615-9880-0/minicluster-data/ts-1/wal/wals/4fe04d5836c745ed9d932f9abbd4574a
I20260621 02:49:16.331995 30221 ts_tablet_manager.cc:1939] T 4fe04d5836c745ed9d932f9abbd4574a P f536f23cf6d64367a1f980eb6a908ea5: Deleting consensus metadata
I20260621 02:49:16.333630 29717 catalog_manager.cc:4841] TS f536f23cf6d64367a1f980eb6a908ea5 (127.9.166.2:36825): tablet 4fe04d5836c745ed9d932f9abbd4574a (table test-table [id=6370a4f548ff4fb7bb72e32690e711bd]) successfully deleted
I20260621 02:49:16.438679 30218 tablet_replica.cc:330] T 4fe04d5836c745ed9d932f9abbd4574a P de3c08fe28594f4f851d0b6f63b46124: stopping tablet replica
I20260621 02:49:16.439580 30218 raft_consensus.cc:2229] T 4fe04d5836c745ed9d932f9abbd4574a P de3c08fe28594f4f851d0b6f63b46124 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260621 02:49:16.440049 30218 raft_consensus.cc:2258] T 4fe04d5836c745ed9d932f9abbd4574a P de3c08fe28594f4f851d0b6f63b46124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260621 02:49:16.440268 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 29789
W20260621 02:49:16.465122 29716 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.166.1:44415 (error 108)
I20260621 02:49:16.465821 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 29920
I20260621 02:49:16.467585 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:16.466652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:44415 (local address 127.0.0.1:59726)
0621 02:49:16.467150 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:16.467313 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:44415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":268,"threads_started":1}
W20260621 02:49:16.467855 29716 catalog_manager.cc:4584] TS de3c08fe28594f4f851d0b6f63b46124 (127.9.166.1:44415): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4fe04d5836c745ed9d932f9abbd4574a: Network error: Client connection negotiation failed: client connection to 127.9.166.1:44415: connect: Connection refused (error 111)
I20260621 02:49:16.473862 30194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:16.466617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.9.166.2:36825)
0621 02:49:16.466808 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:16.466814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:16.467093 (+ 279us) server_negotiation.cc:407] Connection header received
0621 02:49:16.467171 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:16.467176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:16.467299 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:16.467436 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:16.468109 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:16.469257 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:16.470530 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:16.471168 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:16.471288 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:16.472456 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:16.472465 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:16.472479 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:16.473142 (+ 663us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:16.473187 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:16.473213 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:16.473342 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:16.473463 (+ 121us) server_negotiation.cc:299] Negotiation successful
0621 02:49:16.473566 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 02:49:16.474539 30042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:16.521733 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:16.521180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:44415 (local address 127.0.0.1:59730)
0621 02:49:16.521432 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:16.521538 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:44415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260621 02:49:16.522903 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:16.522476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:44415 (local address 127.0.0.1:59742)
0621 02:49:16.522638 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:16.522723 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:44415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260621 02:49:16.563537 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 29920
I20260621 02:49:16.580722 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 30050
I20260621 02:49:16.583492 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:16.582910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:44415 (local address 127.0.0.1:59744)
0621 02:49:16.583166 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:16.583259 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:44415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260621 02:49:16.584760 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:16.584378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:44415 (local address 127.0.0.1:59752)
0621 02:49:16.584532 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:16.584608 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:44415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260621 02:49:16.589473 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:16.581567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48398 (local address 127.9.166.3:38519)
0621 02:49:16.582058 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:16.582062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:16.582077 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:49:16.582136 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:16.582140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:16.582220 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:16.582344 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:16.582806 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:16.584150 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:16.586144 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:16.586582 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:16.586691 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:16.588036 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:16.588044 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:16.588059 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:16.588652 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:16.588696 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:16.588716 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:16.588793 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:16.589013 (+ 220us) server_negotiation.cc:299] Negotiation successful
0621 02:49:16.589086 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391}
I20260621 02:49:16.590273 30172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:16.665254 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:16.664701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:44415 (local address 127.0.0.1:59762)
0621 02:49:16.664956 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:16.665063 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:44415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260621 02:49:16.666416 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:16.665939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:44415 (local address 127.0.0.1:59764)
0621 02:49:16.666153 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:16.666255 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:44415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260621 02:49:16.712186 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 30050
I20260621 02:49:16.732417 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 29699
I20260621 02:49:16.733544 29760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:16.846426 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:16.845768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.1:44415 (local address 127.0.0.1:59780)
0621 02:49:16.846100 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:16.846250 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.1:44415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260621 02:49:16.846499 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 29699
2026-06-21T02:49:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3015 ms)
[----------] 6 tests from EncryptionPolicyTest (21963 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:16 dist-test-slave-hvgn krb5kdc[30230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:16 dist-test-slave-hvgn krb5kdc[30230](info): set up 2 sockets
Jun 21 02:49:16 dist-test-slave-hvgn krb5kdc[30230](info): commencing operation
krb5kdc: starting...
W20260621 02:49:17.971745 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:18 dist-test-slave-hvgn krb5kdc[30230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:49:18.208684 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:18.530471 30243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:18.530808 30243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:18.530963 30243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:18.538218 30243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:18.538374 30243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:18.538477 30243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:18.538583 30243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:18.538666 30243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:18.548576 30243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44889
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:18.549852 30243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:18.551139 30243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:18.563498 30251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:18.563588 30248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:18.563591 30249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:18.564347 30243 server_base.cc:1034] running on GCE node
I20260621 02:49:18.565104 30243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:18.566494 30243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:18.567756 30243 hybrid_clock.cc:648] HybridClock initialized: now 1782010158567762 us; error 106 us; skew 500 ppm
I20260621 02:49:18.568099 30243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:18 dist-test-slave-hvgn krb5kdc[30230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:18.571287 30243 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:49:18.572734 30243 webserver.cc:460] Webserver started at http://127.0.0.1:37415/ using document root <none> and password file <none>
I20260621 02:49:18.573413 30243 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:18.573549 30243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:18.573845 30243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:18.576769 30243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "f8d4bea17fc843c7922b871f4a1cde89"
format_stamp: "Formatted at 2026-06-21 02:49:18 on dist-test-slave-hvgn"
server_key: "aa7bfd60e468db1b1d428e823d9be5f6"
server_key_iv: "3c7c4292466d45cc4118f44da09ba9b1"
server_key_version: "encryptionkey@0"
I20260621 02:49:18.577510 30243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "f8d4bea17fc843c7922b871f4a1cde89"
format_stamp: "Formatted at 2026-06-21 02:49:18 on dist-test-slave-hvgn"
server_key: "aa7bfd60e468db1b1d428e823d9be5f6"
server_key_iv: "3c7c4292466d45cc4118f44da09ba9b1"
server_key_version: "encryptionkey@0"
I20260621 02:49:18.582599 30243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:49:18.585955 30258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:18.587201 30243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:49:18.587419 30243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "f8d4bea17fc843c7922b871f4a1cde89"
format_stamp: "Formatted at 2026-06-21 02:49:18 on dist-test-slave-hvgn"
server_key: "aa7bfd60e468db1b1d428e823d9be5f6"
server_key_iv: "3c7c4292466d45cc4118f44da09ba9b1"
server_key_version: "encryptionkey@0"
I20260621 02:49:18.587617 30243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:18.600525 30243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:18.604615 30243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:18.604897 30243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:18.623086 30243 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44889
I20260621 02:49:18.623107 30309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44889 every 8 connection(s)
I20260621 02:49:18.624447 30243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:18.628165 30310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:18.631146 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 30243
I20260621 02:49:18.631351 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:18.632027 9880 external_mini_cluster.cc:1428] Setting key 8051d74ace42f1313768a4a817b1cfdc
I20260621 02:49:18.640502 30310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89: Bootstrap starting.
I20260621 02:49:18.644277 30310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:18.645617 30310 log.cc:824] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89: Log is configured to *not* fsync() on all Append() calls
Jun 21 02:49:18 dist-test-slave-hvgn krb5kdc[30230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 02:49:18.648613 30310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89: No bootstrap required, opened a new log
I20260621 02:49:18.655354 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:18.634008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51067 (local address 127.0.0.1:44889)
0621 02:49:18.634778 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:18.634789 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:18.634828 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:49:18.636448 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:18.636468 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:18.636865 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:18.637194 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:18.638955 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:18.640337 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:18.642136 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:18.642788 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:18.642908 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:18.648071 (+ 5163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:18.648079 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:18.648092 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:18.648126 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:18.651302 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:18.652115 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:18.652123 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:18.652128 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:18.652296 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:18.652780 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:18.652784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:18.652786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:18.653252 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:18.653586 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:18.654091 (+ 505us) server_negotiation.cc:299] Negotiation successful
0621 02:49:18.654392 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":360,"threads_started":1}
I20260621 02:49:18.656543 30310 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d4bea17fc843c7922b871f4a1cde89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44889 } }
I20260621 02:49:18.657013 30310 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:18.657136 30310 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8d4bea17fc843c7922b871f4a1cde89, State: Initialized, Role: FOLLOWER
I20260621 02:49:18.657837 30310 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d4bea17fc843c7922b871f4a1cde89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44889 } }
I20260621 02:49:18.658118 30310 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:18.658269 30310 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:18.658442 30310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:18.659545 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:49:18.661369 30310 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d4bea17fc843c7922b871f4a1cde89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44889 } }
I20260621 02:49:18.661924 30310 leader_election.cc:302] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8d4bea17fc843c7922b871f4a1cde89; no voters:
I20260621 02:49:18.662581 30310 leader_election.cc:288] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:18.662647 30315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:18.664366 30315 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89 [term 1 LEADER]: Becoming Leader. State: Replica: f8d4bea17fc843c7922b871f4a1cde89, State: Running, Role: LEADER
I20260621 02:49:18.664968 30315 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d4bea17fc843c7922b871f4a1cde89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44889 } }
I20260621 02:49:18.665172 30310 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:18.667464 30318 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8d4bea17fc843c7922b871f4a1cde89. Latest consensus state: current_term: 1 leader_uuid: "f8d4bea17fc843c7922b871f4a1cde89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d4bea17fc843c7922b871f4a1cde89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44889 } } }
I20260621 02:49:18.667870 30318 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:18.668412 30316 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8d4bea17fc843c7922b871f4a1cde89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d4bea17fc843c7922b871f4a1cde89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44889 } } }
I20260621 02:49:18.668712 30316 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:18.671221 30324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:18.683346 30324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:18.709631 30324 catalog_manager.cc:1347] Generated new cluster ID: e380d912a19841acbf6bce34415c56e9
I20260621 02:49:18.709790 30324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:18.712564 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:18.674875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57977 (local address 127.0.0.1:44889)
0621 02:49:18.675115 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:18.675124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:18.688181 (+ 13057us) server_negotiation.cc:407] Connection header received
0621 02:49:18.688453 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:18.688459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:18.688568 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:18.688687 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:18.692106 (+ 3419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:18.693771 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:18.696216 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:18.697157 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:18.697484 (+ 327us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:18.702122 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:18.702132 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:18.702140 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:18.702191 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:18.707101 (+ 4910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:18.708478 (+ 1377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:18.708488 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:18.708494 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:18.708735 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:18.709840 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:18.709850 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:18.709854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:18.710392 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:18.710836 (+ 444us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:18.711758 (+ 922us) server_negotiation.cc:299] Negotiation successful
0621 02:49:18.712014 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":67}
I20260621 02:49:18.726902 30324 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:18.728253 30324 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:18.733250 30324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8d4bea17fc843c7922b871f4a1cde89: Generated new TSK 0
I20260621 02:49:18.734266 30324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:18.744819 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 30243
I20260621 02:49:18.746029 30305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:18.836261 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 30243
W20260621 02:49:18.853451 30228 connection.cc:535] client connection to 127.0.0.1:44889 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44889): Connection reset by peer (error 104)
2026-06-21T02:49:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1984 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:18 dist-test-slave-hvgn krb5kdc[30343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:18 dist-test-slave-hvgn krb5kdc[30343](info): set up 2 sockets
Jun 21 02:49:18 dist-test-slave-hvgn krb5kdc[30343](info): commencing operation
krb5kdc: starting...
W20260621 02:49:19.951767 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:20 dist-test-slave-hvgn krb5kdc[30343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:49:20.188381 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33701
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:20.509492 30356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:20.509829 30356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:20.510004 30356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:20.517022 30356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:20.517182 30356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:20.517290 30356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:20.517400 30356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:20.517482 30356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:20.526953 30356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33701
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:20.528228 30356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:20.529496 30356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:20.541036 30361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:20.540994 30364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:20.541059 30362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:20.541867 30356 server_base.cc:1034] running on GCE node
I20260621 02:49:20.542582 30356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:20.543910 30356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:20.545130 30356 hybrid_clock.cc:648] HybridClock initialized: now 1782010160545088 us; error 29 us; skew 500 ppm
I20260621 02:49:20.545589 30356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:20 dist-test-slave-hvgn krb5kdc[30343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:20.548780 30356 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:49:20.550243 30356 webserver.cc:460] Webserver started at http://127.0.0.1:33479/ using document root <none> and password file <none>
I20260621 02:49:20.550931 30356 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:20.551081 30356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:20.551424 30356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:20.554513 30356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "e4e11843402e4d349ebb631a0847c401"
format_stamp: "Formatted at 2026-06-21 02:49:20 on dist-test-slave-hvgn"
server_key: "7ab83e4b457a6413b762518306e31edb"
server_key_iv: "c3f9f7617b5db030d48dca12070f044f"
server_key_version: "encryptionkey@0"
I20260621 02:49:20.555280 30356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "e4e11843402e4d349ebb631a0847c401"
format_stamp: "Formatted at 2026-06-21 02:49:20 on dist-test-slave-hvgn"
server_key: "7ab83e4b457a6413b762518306e31edb"
server_key_iv: "c3f9f7617b5db030d48dca12070f044f"
server_key_version: "encryptionkey@0"
I20260621 02:49:20.560431 30356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:49:20.563812 30371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:20.565079 30356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:49:20.565260 30356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "e4e11843402e4d349ebb631a0847c401"
format_stamp: "Formatted at 2026-06-21 02:49:20 on dist-test-slave-hvgn"
server_key: "7ab83e4b457a6413b762518306e31edb"
server_key_iv: "c3f9f7617b5db030d48dca12070f044f"
server_key_version: "encryptionkey@0"
I20260621 02:49:20.565451 30356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:20.580124 30356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:20.583913 30356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:20.584190 30356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:20.602341 30356 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33701
I20260621 02:49:20.602365 30422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33701 every 8 connection(s)
I20260621 02:49:20.603714 30356 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:20.607360 30423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:20.609767 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 30356
I20260621 02:49:20.609941 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:20.610459 9880 external_mini_cluster.cc:1428] Setting key 509214616f504e399d487ba92cc934f1
I20260621 02:49:20.621919 30423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401: Bootstrap starting.
I20260621 02:49:20.624634 30426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:20.612278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52497 (local address 127.0.0.1:33701)
0621 02:49:20.613081 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:20.613097 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:20.613146 (+ 49us) server_negotiation.cc:407] Connection header received
0621 02:49:20.614816 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:20.614845 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:20.615353 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:20.615801 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:20.616263 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:20.618169 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:20.619460 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:20.620468 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:20.620653 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:20.621794 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:20.621804 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:20.621821 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:20.622516 (+ 695us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:20.622681 (+ 165us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:20.622713 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:20.622864 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:20.623094 (+ 230us) server_negotiation.cc:299] Negotiation successful
0621 02:49:20.623253 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":322,"threads_started":1}
I20260621 02:49:20.625912 30423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:20.627071 30423 log.cc:824] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:20.628561 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:49:20.630137 30423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401: No bootstrap required, opened a new log
I20260621 02:49:20.640359 30426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:20.633701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35519 (local address 127.0.0.1:33701)
0621 02:49:20.633910 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:20.633917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:20.634227 (+ 310us) server_negotiation.cc:407] Connection header received
0621 02:49:20.634396 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:20.634399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:20.634454 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:20.634534 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:20.635089 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:20.635944 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:20.637841 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:20.638302 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:20.638391 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:20.639591 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:20.639596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:20.639599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:20.639615 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:20.639662 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:20.639676 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:20.639760 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:20.639969 (+ 209us) server_negotiation.cc:299] Negotiation successful
0621 02:49:20.640040 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260621 02:49:20.640434 30423 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e11843402e4d349ebb631a0847c401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33701 } }
I20260621 02:49:20.640877 30423 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:20.641023 30423 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4e11843402e4d349ebb631a0847c401, State: Initialized, Role: FOLLOWER
I20260621 02:49:20.641670 30423 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e11843402e4d349ebb631a0847c401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33701 } }
I20260621 02:49:20.641908 30423 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:20.642051 30423 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:20.642221 30423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:20.644379 30423 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e11843402e4d349ebb631a0847c401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33701 } }
I20260621 02:49:20.644872 30423 leader_election.cc:302] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4e11843402e4d349ebb631a0847c401; no voters:
I20260621 02:49:20.645414 30423 leader_election.cc:288] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:20.645539 30433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:20.646229 30433 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401 [term 1 LEADER]: Becoming Leader. State: Replica: e4e11843402e4d349ebb631a0847c401, State: Running, Role: LEADER
I20260621 02:49:20.646751 30433 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e11843402e4d349ebb631a0847c401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33701 } }
I20260621 02:49:20.647605 30423 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:20.649869 30435 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4e11843402e4d349ebb631a0847c401. Latest consensus state: current_term: 1 leader_uuid: "e4e11843402e4d349ebb631a0847c401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e11843402e4d349ebb631a0847c401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33701 } } }
I20260621 02:49:20.650709 30434 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4e11843402e4d349ebb631a0847c401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e11843402e4d349ebb631a0847c401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33701 } } }
I20260621 02:49:20.650975 30435 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:20.651072 30434 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:20.652676 30443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:20.657938 30443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:20.666509 30443 catalog_manager.cc:1347] Generated new cluster ID: 3383a3cec0e34a3289b4d57c3ea217f1
I20260621 02:49:20.666617 30443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:20.688519 30443 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:20.689781 30443 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:20.694873 30443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4e11843402e4d349ebb631a0847c401: Generated new TSK 0
I20260621 02:49:20.695756 30443 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:20.700503 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 30356
W20260621 02:49:20.701679 30418 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52497
W20260621 02:49:20.702498 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 30356: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:49:20.702708 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 30356
W20260621 02:49:20.716461 30341 connection.cc:535] client connection to 127.0.0.1:33701 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33701): Connection reset by peer (error 104)
2026-06-21T02:49:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1861 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:20 dist-test-slave-hvgn krb5kdc[30454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:20 dist-test-slave-hvgn krb5kdc[30454](info): set up 2 sockets
Jun 21 02:49:20 dist-test-slave-hvgn krb5kdc[30454](info): commencing operation
krb5kdc: starting...
W20260621 02:49:21.801220 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:21 dist-test-slave-hvgn krb5kdc[30454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:49:22.040782 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38601
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:22.360119 30467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:22.360462 30467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:22.360621 30467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:22.367859 30467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:22.368017 30467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:22.368124 30467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:22.368228 30467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:22.368307 30467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:22.378036 30467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38601
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:22.379297 30467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:22.380605 30467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:22.392657 30472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:22.392616 30473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:22.392629 30475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:22.393536 30467 server_base.cc:1034] running on GCE node
I20260621 02:49:22.394333 30467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:22.395710 30467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:22.396976 30467 hybrid_clock.cc:648] HybridClock initialized: now 1782010162396932 us; error 35 us; skew 500 ppm
I20260621 02:49:22.397351 30467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:22 dist-test-slave-hvgn krb5kdc[30454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:22.403848 30467 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:49:22.405259 30467 webserver.cc:460] Webserver started at http://127.0.0.1:42437/ using document root <none> and password file <none>
I20260621 02:49:22.405915 30467 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:22.406011 30467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:22.406252 30467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:22.409296 30467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "24219f8c62a04087805a60b592c4164b"
format_stamp: "Formatted at 2026-06-21 02:49:22 on dist-test-slave-hvgn"
server_key: "22c60c083f7559f73c51615c2f8789f8"
server_key_iv: "e151d5904c60257d0529b6fbedae048a"
server_key_version: "encryptionkey@0"
I20260621 02:49:22.409991 30467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "24219f8c62a04087805a60b592c4164b"
format_stamp: "Formatted at 2026-06-21 02:49:22 on dist-test-slave-hvgn"
server_key: "22c60c083f7559f73c51615c2f8789f8"
server_key_iv: "e151d5904c60257d0529b6fbedae048a"
server_key_version: "encryptionkey@0"
I20260621 02:49:22.414836 30467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:49:22.418102 30482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:22.419360 30467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260621 02:49:22.419611 30467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "24219f8c62a04087805a60b592c4164b"
format_stamp: "Formatted at 2026-06-21 02:49:22 on dist-test-slave-hvgn"
server_key: "22c60c083f7559f73c51615c2f8789f8"
server_key_iv: "e151d5904c60257d0529b6fbedae048a"
server_key_version: "encryptionkey@0"
I20260621 02:49:22.419793 30467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:22.425073 30467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:22.428905 30467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:22.429195 30467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:22.446770 30467 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38601
I20260621 02:49:22.446807 30533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38601 every 8 connection(s)
I20260621 02:49:22.448218 30467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:22.451885 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 30467
I20260621 02:49:22.451925 30534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:22.452059 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:22.452538 9880 external_mini_cluster.cc:1428] Setting key 08ec2622155f73dd167b4b7605ada3d2
I20260621 02:49:22.460232 30537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:22.454184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59873 (local address 127.0.0.1:38601)
0621 02:49:22.455056 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:22.455067 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:22.455125 (+ 58us) server_negotiation.cc:407] Connection header received
0621 02:49:22.456780 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:22.456802 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:22.457159 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:22.457580 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:22.458652 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:22.458658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:22.458670 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:22.458687 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:22.458741 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:22.458760 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:22.458932 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:22.459064 (+ 132us) server_negotiation.cc:299] Negotiation successful
0621 02:49:22.459276 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":407,"threads_started":1}
I20260621 02:49:22.463994 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:49:22.465550 30534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b: Bootstrap starting.
I20260621 02:49:22.469017 30534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:22.470062 30534 log.cc:824] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:22.471100 30537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:22.468508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51201 (local address 127.0.0.1:38601)
0621 02:49:22.468695 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:22.468701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:22.469128 (+ 427us) server_negotiation.cc:407] Connection header received
0621 02:49:22.469335 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:22.469342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:22.469393 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:22.469510 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:22.470438 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:22.470444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:22.470449 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:22.470466 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:22.470521 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:22.470542 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:22.470665 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:22.470769 (+ 104us) server_negotiation.cc:299] Negotiation successful
0621 02:49:22.470831 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260621 02:49:22.473188 30534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b: No bootstrap required, opened a new log
I20260621 02:49:22.480705 30534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24219f8c62a04087805a60b592c4164b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38601 } }
I20260621 02:49:22.481065 30534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:22.481186 30534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24219f8c62a04087805a60b592c4164b, State: Initialized, Role: FOLLOWER
I20260621 02:49:22.481765 30534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24219f8c62a04087805a60b592c4164b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38601 } }
I20260621 02:49:22.482010 30534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:22.482165 30534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:22.482407 30534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:22.484622 30534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24219f8c62a04087805a60b592c4164b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38601 } }
I20260621 02:49:22.485100 30534 leader_election.cc:302] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24219f8c62a04087805a60b592c4164b; no voters:
I20260621 02:49:22.485662 30534 leader_election.cc:288] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:22.485821 30544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:22.486524 30544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b [term 1 LEADER]: Becoming Leader. State: Replica: 24219f8c62a04087805a60b592c4164b, State: Running, Role: LEADER
I20260621 02:49:22.487053 30544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24219f8c62a04087805a60b592c4164b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38601 } }
I20260621 02:49:22.487739 30534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:22.489768 30546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24219f8c62a04087805a60b592c4164b. Latest consensus state: current_term: 1 leader_uuid: "24219f8c62a04087805a60b592c4164b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24219f8c62a04087805a60b592c4164b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38601 } } }
I20260621 02:49:22.490198 30546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:22.490088 30545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24219f8c62a04087805a60b592c4164b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24219f8c62a04087805a60b592c4164b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38601 } } }
I20260621 02:49:22.490361 30545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:22.491537 30551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:22.495873 30551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:22.504681 30551 catalog_manager.cc:1347] Generated new cluster ID: b3ee4d13ecee4c299a7932b1928639e1
I20260621 02:49:22.504797 30551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:22.522949 30551 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:22.523479 30551 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:22.530961 30551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24219f8c62a04087805a60b592c4164b: Generated new TSK 0
I20260621 02:49:22.531780 30551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:22.543649 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 30467
W20260621 02:49:22.544902 30529 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59873
W20260621 02:49:22.545710 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 30467: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:49:22.545892 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 30467
2026-06-21T02:49:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1846 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:22 dist-test-slave-hvgn krb5kdc[30565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:22 dist-test-slave-hvgn krb5kdc[30565](info): set up 2 sockets
Jun 21 02:49:22 dist-test-slave-hvgn krb5kdc[30565](info): commencing operation
krb5kdc: starting...
W20260621 02:49:23.647782 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:23 dist-test-slave-hvgn krb5kdc[30565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:49:23.886528 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40165
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:24.217021 30578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:24.217367 30578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:24.217530 30578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:24.224561 30578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:24.224723 30578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:24.224834 30578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:24.224944 30578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:24.225031 30578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:24.234766 30578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:24.236104 30578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:24.237439 30578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:24.248977 30586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:24.249008 30583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:24.249186 30584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:24.250252 30578 server_base.cc:1034] running on GCE node
I20260621 02:49:24.251025 30578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:24.252470 30578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:24.253675 30578 hybrid_clock.cc:648] HybridClock initialized: now 1782010164253636 us; error 30 us; skew 500 ppm
I20260621 02:49:24.254030 30578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:24 dist-test-slave-hvgn krb5kdc[30565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:24.257326 30578 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:49:24.258941 30578 webserver.cc:460] Webserver started at http://127.0.0.1:38537/ using document root <none> and password file <none>
I20260621 02:49:24.259744 30578 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:24.259878 30578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:24.260188 30578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:24.263195 30578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "730d1401686f4044aed8e44906b975e8"
format_stamp: "Formatted at 2026-06-21 02:49:24 on dist-test-slave-hvgn"
server_key: "c26c4b3b8fe0111a7d5fdb9ad4e54248"
server_key_iv: "6b8b7e5ef960613a67603454fc142ad0"
server_key_version: "encryptionkey@0"
I20260621 02:49:24.264038 30578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "730d1401686f4044aed8e44906b975e8"
format_stamp: "Formatted at 2026-06-21 02:49:24 on dist-test-slave-hvgn"
server_key: "c26c4b3b8fe0111a7d5fdb9ad4e54248"
server_key_iv: "6b8b7e5ef960613a67603454fc142ad0"
server_key_version: "encryptionkey@0"
I20260621 02:49:24.269271 30578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:49:24.272624 30593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:24.273969 30578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:49:24.274158 30578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "730d1401686f4044aed8e44906b975e8"
format_stamp: "Formatted at 2026-06-21 02:49:24 on dist-test-slave-hvgn"
server_key: "c26c4b3b8fe0111a7d5fdb9ad4e54248"
server_key_iv: "6b8b7e5ef960613a67603454fc142ad0"
server_key_version: "encryptionkey@0"
I20260621 02:49:24.274350 30578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:24.307523 30578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:24.311614 30578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:24.311970 30578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:24.330470 30578 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40165
I20260621 02:49:24.330494 30644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40165 every 8 connection(s)
I20260621 02:49:24.331918 30578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:24.335685 30645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:24.340339 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 30578
I20260621 02:49:24.340541 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:24.341084 9880 external_mini_cluster.cc:1428] Setting key e8466111a5ca3b305775f1b0fecf6862
I20260621 02:49:24.347129 30645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8: Bootstrap starting.
I20260621 02:49:24.350762 30645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:24.351990 30645 log.cc:824] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:24.354718 30645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8: No bootstrap required, opened a new log
Jun 21 02:49:24 dist-test-slave-hvgn krb5kdc[30565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 02:49:24.363180 30645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730d1401686f4044aed8e44906b975e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40165 } }
I20260621 02:49:24.363621 30645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:24.363750 30645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730d1401686f4044aed8e44906b975e8, State: Initialized, Role: FOLLOWER
I20260621 02:49:24.364081 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:24.342871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35053 (local address 127.0.0.1:40165)
0621 02:49:24.343674 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:24.343686 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:24.343727 (+ 41us) server_negotiation.cc:407] Connection header received
0621 02:49:24.345621 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:24.345645 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:24.346041 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:24.346359 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:24.348173 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:24.349513 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:24.351335 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:24.352066 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:24.352268 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:24.356468 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:24.356478 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:24.356492 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:24.356528 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:24.359750 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:24.360451 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:24.360458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:24.360465 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:24.360637 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:24.361032 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:24.361037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:24.361042 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:24.361585 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:24.361924 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:24.362404 (+ 480us) server_negotiation.cc:299] Negotiation successful
0621 02:49:24.362837 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":365,"threads_started":1}
I20260621 02:49:24.364415 30645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730d1401686f4044aed8e44906b975e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40165 } }
I20260621 02:49:24.364678 30645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:24.364835 30645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:24.365054 30645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:24.367583 30645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730d1401686f4044aed8e44906b975e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40165 } }
I20260621 02:49:24.367883 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:49:24.368095 30645 leader_election.cc:302] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 730d1401686f4044aed8e44906b975e8; no voters:
I20260621 02:49:24.368608 30645 leader_election.cc:288] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:24.368750 30650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:24.370647 30645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:24.375838 30650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8 [term 1 LEADER]: Becoming Leader. State: Replica: 730d1401686f4044aed8e44906b975e8, State: Running, Role: LEADER
I20260621 02:49:24.376523 30650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730d1401686f4044aed8e44906b975e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40165 } }
I20260621 02:49:24.380141 30651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "730d1401686f4044aed8e44906b975e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730d1401686f4044aed8e44906b975e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40165 } } }
I20260621 02:49:24.380542 30651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:24.381011 30652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 730d1401686f4044aed8e44906b975e8. Latest consensus state: current_term: 1 leader_uuid: "730d1401686f4044aed8e44906b975e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730d1401686f4044aed8e44906b975e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40165 } } }
I20260621 02:49:24.381242 30652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:24.384516 30664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:24.392233 30664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:24.394661 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:24.374613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32787 (local address 127.0.0.1:40165)
0621 02:49:24.374834 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:24.374843 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:24.375466 (+ 623us) server_negotiation.cc:407] Connection header received
0621 02:49:24.375791 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:24.375800 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:24.375907 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:24.376017 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:24.378609 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:24.380479 (+ 1870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:24.382880 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:24.383864 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:24.384019 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:24.386711 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:24.386716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:24.386720 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:24.386753 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:24.390587 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:24.391558 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:24.391565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:24.391571 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:24.391716 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:24.392175 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:24.392181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:24.392184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:24.392637 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:24.392852 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:24.394017 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0621 02:49:24.394249 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260621 02:49:24.404639 30664 catalog_manager.cc:1347] Generated new cluster ID: 94ca604ab8404b978436c2b6973f7496
I20260621 02:49:24.404778 30664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:24.415588 30664 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:24.416862 30664 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:24.429174 30664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 730d1401686f4044aed8e44906b975e8: Generated new TSK 0
I20260621 02:49:24.430078 30664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:24.444375 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 30578
I20260621 02:49:24.445422 30640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:24.538867 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 30578
2026-06-21T02:49:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1992 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:24 dist-test-slave-hvgn krb5kdc[30677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:24 dist-test-slave-hvgn krb5kdc[30677](info): set up 2 sockets
Jun 21 02:49:24 dist-test-slave-hvgn krb5kdc[30677](info): commencing operation
krb5kdc: starting...
W20260621 02:49:25.703966 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:25 dist-test-slave-hvgn krb5kdc[30677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:49:25.948371 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37733
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:26.271492 30690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:26.271831 30690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:26.271987 30690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:26.279156 30690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:26.279304 30690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:26.279434 30690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:26.279544 30690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:26.279623 30690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:26.289381 30690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:26.290715 30690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:26.292027 30690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:26.303512 30695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:26.303556 30696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:26.303802 30698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:26.304690 30690 server_base.cc:1034] running on GCE node
I20260621 02:49:26.305511 30690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:26.306929 30690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:26.308154 30690 hybrid_clock.cc:648] HybridClock initialized: now 1782010166308109 us; error 38 us; skew 500 ppm
I20260621 02:49:26.308501 30690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:26 dist-test-slave-hvgn krb5kdc[30677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:26.311663 30690 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:49:26.313048 30690 webserver.cc:460] Webserver started at http://127.0.0.1:46223/ using document root <none> and password file <none>
I20260621 02:49:26.313730 30690 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:26.313870 30690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:26.314172 30690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:26.317248 30690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "a3232216724944c79d9ade6813b38a2e"
format_stamp: "Formatted at 2026-06-21 02:49:26 on dist-test-slave-hvgn"
server_key: "43fb40375f8bf1f578b1bce458da2360"
server_key_iv: "fb2496ffc571858a836f9d48943daa5b"
server_key_version: "encryptionkey@0"
I20260621 02:49:26.318017 30690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "a3232216724944c79d9ade6813b38a2e"
format_stamp: "Formatted at 2026-06-21 02:49:26 on dist-test-slave-hvgn"
server_key: "43fb40375f8bf1f578b1bce458da2360"
server_key_iv: "fb2496ffc571858a836f9d48943daa5b"
server_key_version: "encryptionkey@0"
I20260621 02:49:26.323225 30690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260621 02:49:26.326828 30705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:26.328218 30690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:49:26.328402 30690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "a3232216724944c79d9ade6813b38a2e"
format_stamp: "Formatted at 2026-06-21 02:49:26 on dist-test-slave-hvgn"
server_key: "43fb40375f8bf1f578b1bce458da2360"
server_key_iv: "fb2496ffc571858a836f9d48943daa5b"
server_key_version: "encryptionkey@0"
I20260621 02:49:26.328599 30690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:26.364243 30690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:26.368433 30690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:26.368708 30690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:26.388195 30690 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37733
I20260621 02:49:26.388237 30756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37733 every 8 connection(s)
I20260621 02:49:26.389600 30690 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:26.391160 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 30690
I20260621 02:49:26.391335 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:26.391947 9880 external_mini_cluster.cc:1428] Setting key 69d16a1d75a1dbdf529b96ce72f0094a
I20260621 02:49:26.394068 30757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:26.404922 30760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:26.393888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35883 (local address 127.0.0.1:37733)
0621 02:49:26.394778 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:26.394790 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:26.394832 (+ 42us) server_negotiation.cc:407] Connection header received
0621 02:49:26.396507 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:26.396530 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:26.396924 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:26.397346 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:26.397899 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:26.399248 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:26.400611 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:26.401272 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:26.401484 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:26.402690 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:26.402697 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:26.402715 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:26.403277 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:26.403363 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:26.403430 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:26.403561 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:26.403770 (+ 209us) server_negotiation.cc:299] Negotiation successful
0621 02:49:26.404020 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":453,"threads_started":1}
I20260621 02:49:26.405275 30757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e: Bootstrap starting.
I20260621 02:49:26.408500 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:49:26.408527 30757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:26.409458 30757 log.cc:824] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:26.412108 30757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e: No bootstrap required, opened a new log
I20260621 02:49:26.419734 30760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:26.413126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58949 (local address 127.0.0.1:37733)
0621 02:49:26.413294 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:26.413301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:26.413767 (+ 466us) server_negotiation.cc:407] Connection header received
0621 02:49:26.413907 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:26.413913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:26.413988 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:26.414095 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:26.414584 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:26.415737 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:26.417000 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:26.417738 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:26.417873 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:26.418881 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:26.418887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:26.418892 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:26.418910 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:26.418953 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:26.418972 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:26.419059 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:26.419233 (+ 174us) server_negotiation.cc:299] Negotiation successful
0621 02:49:26.419332 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:49:26.421541 30757 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3232216724944c79d9ade6813b38a2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37733 } }
I20260621 02:49:26.421885 30757 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:26.422003 30757 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3232216724944c79d9ade6813b38a2e, State: Initialized, Role: FOLLOWER
I20260621 02:49:26.422559 30757 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3232216724944c79d9ade6813b38a2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37733 } }
I20260621 02:49:26.422780 30757 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:26.422901 30757 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:26.423132 30757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:26.425444 30757 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3232216724944c79d9ade6813b38a2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37733 } }
I20260621 02:49:26.426045 30757 leader_election.cc:302] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3232216724944c79d9ade6813b38a2e; no voters:
I20260621 02:49:26.426766 30757 leader_election.cc:288] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:26.426855 30767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:26.427758 30767 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e [term 1 LEADER]: Becoming Leader. State: Replica: a3232216724944c79d9ade6813b38a2e, State: Running, Role: LEADER
I20260621 02:49:26.428256 30767 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3232216724944c79d9ade6813b38a2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37733 } }
I20260621 02:49:26.428956 30757 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:26.431144 30769 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3232216724944c79d9ade6813b38a2e. Latest consensus state: current_term: 1 leader_uuid: "a3232216724944c79d9ade6813b38a2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3232216724944c79d9ade6813b38a2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37733 } } }
I20260621 02:49:26.431324 30768 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3232216724944c79d9ade6813b38a2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3232216724944c79d9ade6813b38a2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37733 } } }
I20260621 02:49:26.431607 30769 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:26.431646 30768 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:26.434152 30776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:26.439260 30776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:26.448377 30776 catalog_manager.cc:1347] Generated new cluster ID: 6f0c82e7bfc64e98a4c2fdf3cab836af
I20260621 02:49:26.448487 30776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:26.498657 30776 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:26.500066 30776 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:26.506960 30776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3232216724944c79d9ade6813b38a2e: Generated new TSK 0
I20260621 02:49:26.507823 30776 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:26.513803 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 30690
W20260621 02:49:26.515257 30752 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35883
W20260621 02:49:26.515964 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 30690: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:49:26.516124 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 30690
2026-06-21T02:49:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1974 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:26 dist-test-slave-hvgn krb5kdc[30788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:26 dist-test-slave-hvgn krb5kdc[30788](info): set up 2 sockets
Jun 21 02:49:26 dist-test-slave-hvgn krb5kdc[30788](info): commencing operation
krb5kdc: starting...
W20260621 02:49:27.623908 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:27 dist-test-slave-hvgn krb5kdc[30788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:49:27.864711 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:28.187629 30801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:28.187914 30801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:28.188017 30801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:28.194988 30801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:28.195112 30801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:28.195169 30801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:28.195225 30801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:28.195267 30801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:28.204779 30801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:28.205961 30801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:28.207185 30801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:28.218425 30806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:28.218509 30807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:28.218472 30809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:28.219596 30801 server_base.cc:1034] running on GCE node
I20260621 02:49:28.220425 30801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:28.221866 30801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:28.223080 30801 hybrid_clock.cc:648] HybridClock initialized: now 1782010168223024 us; error 49 us; skew 500 ppm
I20260621 02:49:28.223485 30801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:28 dist-test-slave-hvgn krb5kdc[30788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:28.233582 30801 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:49:28.235077 30801 webserver.cc:460] Webserver started at http://127.0.0.1:44651/ using document root <none> and password file <none>
I20260621 02:49:28.235852 30801 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:28.235997 30801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:28.236306 30801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:28.239295 30801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "0770061e65f64319a3b15e15af0810ad"
format_stamp: "Formatted at 2026-06-21 02:49:28 on dist-test-slave-hvgn"
server_key: "86bf26523541e9678ba340e97f595241"
server_key_iv: "8e8d94c1fd4847496bbaecdc80be802c"
server_key_version: "encryptionkey@0"
I20260621 02:49:28.240103 30801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "0770061e65f64319a3b15e15af0810ad"
format_stamp: "Formatted at 2026-06-21 02:49:28 on dist-test-slave-hvgn"
server_key: "86bf26523541e9678ba340e97f595241"
server_key_iv: "8e8d94c1fd4847496bbaecdc80be802c"
server_key_version: "encryptionkey@0"
I20260621 02:49:28.245123 30801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:49:28.248548 30816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:28.249806 30801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:49:28.250001 30801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "0770061e65f64319a3b15e15af0810ad"
format_stamp: "Formatted at 2026-06-21 02:49:28 on dist-test-slave-hvgn"
server_key: "86bf26523541e9678ba340e97f595241"
server_key_iv: "8e8d94c1fd4847496bbaecdc80be802c"
server_key_version: "encryptionkey@0"
I20260621 02:49:28.250185 30801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:28.255815 30801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:28.259883 30801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:28.260169 30801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:28.277930 30801 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45305
I20260621 02:49:28.277966 30867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45305 every 8 connection(s)
I20260621 02:49:28.279278 30801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:28.282969 30868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:28.287111 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 30801
I20260621 02:49:28.287298 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:28.287748 9880 external_mini_cluster.cc:1428] Setting key ac950c781f6bc34da1896ac35573786b
I20260621 02:49:28.294875 30868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad: Bootstrap starting.
I20260621 02:49:28.296690 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:28.289490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51691 (local address 127.0.0.1:45305)
0621 02:49:28.290263 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:28.290278 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:28.290331 (+ 53us) server_negotiation.cc:407] Connection header received
0621 02:49:28.293058 (+ 2727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:28.293085 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:28.293522 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:28.293922 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:28.294849 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:28.294863 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:28.294880 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:28.294900 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:28.294967 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:28.294996 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:28.295203 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:28.295322 (+ 119us) server_negotiation.cc:299] Negotiation successful
0621 02:49:28.295527 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":325,"threads_started":1}
I20260621 02:49:28.298300 30868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:28.299479 30868 log.cc:824] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:28.300227 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:49:28.302415 30868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad: No bootstrap required, opened a new log
I20260621 02:49:28.307524 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:28.305008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51175 (local address 127.0.0.1:45305)
0621 02:49:28.305221 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:28.305230 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:28.305666 (+ 436us) server_negotiation.cc:407] Connection header received
0621 02:49:28.305815 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:28.305818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:28.305860 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:28.305946 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:28.306855 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:28.306859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:28.306862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:28.306879 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:28.306917 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:28.306931 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:28.307015 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:28.307142 (+ 127us) server_negotiation.cc:299] Negotiation successful
0621 02:49:28.307183 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 02:49:28.311641 30868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0770061e65f64319a3b15e15af0810ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45305 } }
I20260621 02:49:28.311993 30868 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:28.312117 30868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0770061e65f64319a3b15e15af0810ad, State: Initialized, Role: FOLLOWER
I20260621 02:49:28.312652 30868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0770061e65f64319a3b15e15af0810ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45305 } }
I20260621 02:49:28.312877 30868 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:28.313038 30868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:28.313226 30868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:28.315596 30868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0770061e65f64319a3b15e15af0810ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45305 } }
I20260621 02:49:28.316048 30868 leader_election.cc:302] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0770061e65f64319a3b15e15af0810ad; no voters:
I20260621 02:49:28.316538 30868 leader_election.cc:288] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:28.316658 30878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:28.317760 30878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad [term 1 LEADER]: Becoming Leader. State: Replica: 0770061e65f64319a3b15e15af0810ad, State: Running, Role: LEADER
I20260621 02:49:28.318257 30878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0770061e65f64319a3b15e15af0810ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45305 } }
I20260621 02:49:28.318475 30868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:28.321478 30879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0770061e65f64319a3b15e15af0810ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0770061e65f64319a3b15e15af0810ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45305 } } }
I20260621 02:49:28.321435 30880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0770061e65f64319a3b15e15af0810ad. Latest consensus state: current_term: 1 leader_uuid: "0770061e65f64319a3b15e15af0810ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0770061e65f64319a3b15e15af0810ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45305 } } }
I20260621 02:49:28.321897 30879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:28.321931 30880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:28.324175 30887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:28.329146 30887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:28.337390 30887 catalog_manager.cc:1347] Generated new cluster ID: 300ac047c3c745efab1df013975576c9
I20260621 02:49:28.337502 30887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:28.360555 30887 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:28.360960 30887 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:28.373764 30887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0770061e65f64319a3b15e15af0810ad: Generated new TSK 0
I20260621 02:49:28.374612 30887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:28.380515 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 30801
W20260621 02:49:28.381836 30863 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51691
W20260621 02:49:28.382596 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 30801: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:49:28.382778 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 30801
2026-06-21T02:49:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1868 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:28 dist-test-slave-hvgn krb5kdc[30899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:28 dist-test-slave-hvgn krb5kdc[30899](info): set up 2 sockets
Jun 21 02:49:28 dist-test-slave-hvgn krb5kdc[30899](info): commencing operation
krb5kdc: starting...
W20260621 02:49:29.487788 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:29 dist-test-slave-hvgn krb5kdc[30899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:49:29.732375 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:30.053866 30912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:30.054137 30912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:30.054320 30912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:30.061462 30912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:30.061591 30912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:30.061677 30912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:30.061770 30912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:30.061851 30912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:30.071601 30912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:30.072861 30912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:30.074138 30912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:30.085476 30920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:30.085529 30918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:30.085449 30917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:30.086681 30912 server_base.cc:1034] running on GCE node
I20260621 02:49:30.087453 30912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:30.088698 30912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:30.089891 30912 hybrid_clock.cc:648] HybridClock initialized: now 1782010170089857 us; error 32 us; skew 500 ppm
I20260621 02:49:30.090174 30912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:30 dist-test-slave-hvgn krb5kdc[30899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:30.093282 30912 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:49:30.094758 30912 webserver.cc:460] Webserver started at http://127.0.0.1:36091/ using document root <none> and password file <none>
I20260621 02:49:30.095543 30912 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:30.095681 30912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:30.096009 30912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:30.099159 30912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "926067589a4544aa8716b2452739981f"
format_stamp: "Formatted at 2026-06-21 02:49:30 on dist-test-slave-hvgn"
server_key: "24161da7ff2171e58f1301627395065a"
server_key_iv: "71f521d6f39f1f1634aa2016ea24c3e0"
server_key_version: "encryptionkey@0"
I20260621 02:49:30.100029 30912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "926067589a4544aa8716b2452739981f"
format_stamp: "Formatted at 2026-06-21 02:49:30 on dist-test-slave-hvgn"
server_key: "24161da7ff2171e58f1301627395065a"
server_key_iv: "71f521d6f39f1f1634aa2016ea24c3e0"
server_key_version: "encryptionkey@0"
I20260621 02:49:30.105261 30912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260621 02:49:30.108878 30927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:30.110190 30912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260621 02:49:30.110384 30912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "926067589a4544aa8716b2452739981f"
format_stamp: "Formatted at 2026-06-21 02:49:30 on dist-test-slave-hvgn"
server_key: "24161da7ff2171e58f1301627395065a"
server_key_iv: "71f521d6f39f1f1634aa2016ea24c3e0"
server_key_version: "encryptionkey@0"
I20260621 02:49:30.110591 30912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:30.129973 30912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:30.134305 30912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:30.134622 30912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:30.153647 30912 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44135
I20260621 02:49:30.153685 30978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44135 every 8 connection(s)
I20260621 02:49:30.155093 30912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:30.155781 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 30912
I20260621 02:49:30.155967 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:30.156543 9880 external_mini_cluster.cc:1428] Setting key 0e3c378dd50b5bcfa5392b4859bf2c70
I20260621 02:49:30.159528 30979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:49:30 dist-test-slave-hvgn krb5kdc[30899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 02:49:30.172374 30979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f: Bootstrap starting.
I20260621 02:49:30.175490 30979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:30.176487 30979 log.cc:824] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:30.179512 30979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f: No bootstrap required, opened a new log
I20260621 02:49:30.179622 30982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:30.158409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:39913 (local address 127.0.0.1:44135)
0621 02:49:30.159148 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:30.159160 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:30.159200 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:49:30.160913 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:30.160934 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:30.161291 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:30.161601 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:30.163357 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:30.164792 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:30.166600 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:30.167225 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:30.167363 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:30.172137 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:30.172145 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:30.172158 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:30.172195 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:30.175165 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:30.176042 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:30.176048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:30.176054 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:30.176216 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:30.176739 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:30.176743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:30.176745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:30.177441 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:30.177797 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:30.178248 (+ 451us) server_negotiation.cc:299] Negotiation successful
0621 02:49:30.178578 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":348,"threads_started":1}
I20260621 02:49:30.183528 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:49:30.189685 30979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "926067589a4544aa8716b2452739981f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } }
I20260621 02:49:30.190037 30979 raft_consensus.cc:374] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:30.190128 30979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926067589a4544aa8716b2452739981f, State: Initialized, Role: FOLLOWER
I20260621 02:49:30.190696 30979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "926067589a4544aa8716b2452739981f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } }
I20260621 02:49:30.190918 30979 raft_consensus.cc:388] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:30.191057 30979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:30.191274 30979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:30.193827 30979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "926067589a4544aa8716b2452739981f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } }
I20260621 02:49:30.194440 30979 leader_election.cc:302] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 926067589a4544aa8716b2452739981f; no voters:
I20260621 02:49:30.195120 30979 leader_election.cc:288] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:30.195220 30989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:30.197091 30979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:30.197955 30989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f [term 1 LEADER]: Becoming Leader. State: Replica: 926067589a4544aa8716b2452739981f, State: Running, Role: LEADER
I20260621 02:49:30.198669 30989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "926067589a4544aa8716b2452739981f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } }
I20260621 02:49:30.201761 30991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 926067589a4544aa8716b2452739981f. Latest consensus state: current_term: 1 leader_uuid: "926067589a4544aa8716b2452739981f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "926067589a4544aa8716b2452739981f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } } }
I20260621 02:49:30.202105 30991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:30.206507 30982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:30.188665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:59661 (local address 127.0.0.1:44135)
0621 02:49:30.188855 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:30.188860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:30.189216 (+ 356us) server_negotiation.cc:407] Connection header received
0621 02:49:30.189341 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:30.189346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:30.189417 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:30.189525 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:30.191026 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:30.192224 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:30.193448 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:30.194126 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:30.194254 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:30.197652 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:30.197658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:30.197662 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:30.197701 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:30.202199 (+ 4498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:30.203486 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:30.203494 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:30.203500 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:30.203698 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:30.204361 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:30.204378 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:30.204384 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:30.204891 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:30.205253 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:30.205743 (+ 490us) server_negotiation.cc:299] Negotiation successful
0621 02:49:30.205976 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:49:30.207486 30990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "926067589a4544aa8716b2452739981f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "926067589a4544aa8716b2452739981f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44135 } } }
I20260621 02:49:30.207772 30990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:30.212064 30999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:30.217371 30999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:30.225457 30999 catalog_manager.cc:1347] Generated new cluster ID: 3f1ed4f1e9af47feb3b433e664088961
I20260621 02:49:30.225567 30999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:30.233708 30999 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:30.234925 30999 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:30.245021 30999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 926067589a4544aa8716b2452739981f: Generated new TSK 0
I20260621 02:49:30.245957 30999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:30.257922 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 30912
I20260621 02:49:30.258988 30974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:30.348670 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 30912
W20260621 02:49:30.362699 30897 connection.cc:535] client connection to 127.0.0.1:44135 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44135): Connection reset by peer (error 104)
2026-06-21T02:49:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1962 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:30 dist-test-slave-hvgn krb5kdc[31011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:30 dist-test-slave-hvgn krb5kdc[31011](info): set up 2 sockets
Jun 21 02:49:30 dist-test-slave-hvgn krb5kdc[31011](info): commencing operation
krb5kdc: starting...
W20260621 02:49:31.443696 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:31 dist-test-slave-hvgn krb5kdc[31011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:49:31.680482 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46481
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:32.002852 31024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:32.003129 31024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:32.003295 31024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:32.010365 31024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:32.010469 31024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:32.010581 31024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:32.010679 31024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:32.010757 31024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:32.020161 31024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:32.021394 31024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:32.022670 31024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:32.034157 31029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:32.034193 31030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:32.034215 31032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:32.035434 31024 server_base.cc:1034] running on GCE node
I20260621 02:49:32.036157 31024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:32.037500 31024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:32.038697 31024 hybrid_clock.cc:648] HybridClock initialized: now 1782010172038653 us; error 38 us; skew 500 ppm
I20260621 02:49:32.039031 31024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:32 dist-test-slave-hvgn krb5kdc[31011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:32.041991 31024 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:49:32.043552 31024 webserver.cc:460] Webserver started at http://127.0.0.1:40249/ using document root <none> and password file <none>
I20260621 02:49:32.044238 31024 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:32.044384 31024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:32.044679 31024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:32.047601 31024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "3dd2505b443a4572951e6f0f423c2a5c"
format_stamp: "Formatted at 2026-06-21 02:49:32 on dist-test-slave-hvgn"
server_key: "3b8f325ee748e27f4e2922978cb042c1"
server_key_iv: "f2b841d288d7de9a0da1f7b65e1152c1"
server_key_version: "encryptionkey@0"
I20260621 02:49:32.048345 31024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "3dd2505b443a4572951e6f0f423c2a5c"
format_stamp: "Formatted at 2026-06-21 02:49:32 on dist-test-slave-hvgn"
server_key: "3b8f325ee748e27f4e2922978cb042c1"
server_key_iv: "f2b841d288d7de9a0da1f7b65e1152c1"
server_key_version: "encryptionkey@0"
I20260621 02:49:32.053189 31024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:49:32.056449 31039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:32.057924 31024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:49:32.058110 31024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "3dd2505b443a4572951e6f0f423c2a5c"
format_stamp: "Formatted at 2026-06-21 02:49:32 on dist-test-slave-hvgn"
server_key: "3b8f325ee748e27f4e2922978cb042c1"
server_key_iv: "f2b841d288d7de9a0da1f7b65e1152c1"
server_key_version: "encryptionkey@0"
I20260621 02:49:32.058305 31024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:32.084688 31024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:32.088581 31024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:32.088856 31024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:32.106922 31024 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46481
I20260621 02:49:32.106961 31090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46481 every 8 connection(s)
I20260621 02:49:32.108321 31024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:32.111533 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 31024
I20260621 02:49:32.111709 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:32.112139 31091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:32.112159 9880 external_mini_cluster.cc:1428] Setting key 11a51874cd62c855640308bda69a68eb
I20260621 02:49:32.127041 31094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:32.113969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:41545 (local address 127.0.0.1:46481)
0621 02:49:32.114772 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:32.114786 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:32.114839 (+ 53us) server_negotiation.cc:407] Connection header received
0621 02:49:32.116990 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:32.117020 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:32.117555 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:32.117979 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:32.118522 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:32.120542 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:32.121872 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:32.122930 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:32.123115 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:32.124317 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:32.124328 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:32.124345 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:32.125074 (+ 729us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:32.125166 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:32.125197 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:32.125363 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:32.125562 (+ 199us) server_negotiation.cc:299] Negotiation successful
0621 02:49:32.125725 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":313,"threads_started":1}
I20260621 02:49:32.127278 31091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c: Bootstrap starting.
I20260621 02:49:32.130476 31091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:32.130838 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:49:32.131641 31091 log.cc:824] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:32.134368 31091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c: No bootstrap required, opened a new log
I20260621 02:49:32.142187 31094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:32.135473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:58123 (local address 127.0.0.1:46481)
0621 02:49:32.135675 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:32.135682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:32.136068 (+ 386us) server_negotiation.cc:407] Connection header received
0621 02:49:32.136229 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:32.136233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:32.136296 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:32.136376 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:32.136935 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:32.137715 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:32.139713 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:32.140163 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:32.140254 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:32.141429 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:32.141436 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:32.141439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:32.141455 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:32.141493 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:32.141507 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:32.141583 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:32.141773 (+ 190us) server_negotiation.cc:299] Negotiation successful
0621 02:49:32.141835 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260621 02:49:32.144332 31091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd2505b443a4572951e6f0f423c2a5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260621 02:49:32.144696 31091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:32.144809 31091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd2505b443a4572951e6f0f423c2a5c, State: Initialized, Role: FOLLOWER
I20260621 02:49:32.145340 31091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd2505b443a4572951e6f0f423c2a5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260621 02:49:32.145572 31091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:32.145715 31091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:32.145880 31091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:32.148164 31091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd2505b443a4572951e6f0f423c2a5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260621 02:49:32.148785 31091 leader_election.cc:302] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dd2505b443a4572951e6f0f423c2a5c; no voters:
I20260621 02:49:32.149341 31091 leader_election.cc:288] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:32.149484 31101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:32.150182 31101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c [term 1 LEADER]: Becoming Leader. State: Replica: 3dd2505b443a4572951e6f0f423c2a5c, State: Running, Role: LEADER
I20260621 02:49:32.150701 31101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd2505b443a4572951e6f0f423c2a5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } }
I20260621 02:49:32.151341 31091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:32.153326 31102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dd2505b443a4572951e6f0f423c2a5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd2505b443a4572951e6f0f423c2a5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } } }
I20260621 02:49:32.153728 31102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:32.153803 31103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dd2505b443a4572951e6f0f423c2a5c. Latest consensus state: current_term: 1 leader_uuid: "3dd2505b443a4572951e6f0f423c2a5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd2505b443a4572951e6f0f423c2a5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46481 } } }
I20260621 02:49:32.154101 31103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:32.156641 31106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:32.161871 31106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:32.170610 31106 catalog_manager.cc:1347] Generated new cluster ID: ecbf0237eb8a4e2f87354c64a3297a03
I20260621 02:49:32.170715 31106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:32.203460 31106 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:32.205379 31106 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:32.211612 31106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dd2505b443a4572951e6f0f423c2a5c: Generated new TSK 0
I20260621 02:49:32.212613 31106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:32.224927 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 31024
W20260621 02:49:32.225998 31086 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.117:41545
W20260621 02:49:32.226724 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 31024: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:49:32.226869 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 31024
W20260621 02:49:32.240871 31009 connection.cc:535] client connection to 127.0.0.1:46481 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46481): Connection reset by peer (error 104)
2026-06-21T02:49:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1878 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:32 dist-test-slave-hvgn krb5kdc[31122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:32 dist-test-slave-hvgn krb5kdc[31122](info): set up 2 sockets
Jun 21 02:49:32 dist-test-slave-hvgn krb5kdc[31122](info): commencing operation
krb5kdc: starting...
W20260621 02:49:33.323824 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:33 dist-test-slave-hvgn krb5kdc[31122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:49:33.562332 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44047
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:33.886730 31135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:33.887032 31135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:33.887213 31135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:33.894577 31135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:33.894713 31135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:33.894809 31135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:33.894907 31135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:33.894973 31135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:33.904995 31135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:33.906363 31135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:33.907771 31135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:33.919080 31140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:33.919101 31143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:33.919274 31141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:33.919931 31135 server_base.cc:1034] running on GCE node
I20260621 02:49:33.920674 31135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:33.922021 31135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:33.923302 31135 hybrid_clock.cc:648] HybridClock initialized: now 1782010173923216 us; error 46 us; skew 500 ppm
I20260621 02:49:33.923676 31135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:33 dist-test-slave-hvgn krb5kdc[31122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:33.929893 31135 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:49:33.931444 31135 webserver.cc:460] Webserver started at http://127.0.0.1:43497/ using document root <none> and password file <none>
I20260621 02:49:33.932121 31135 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:33.932257 31135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:33.932554 31135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:33.935563 31135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "6eae4e68188343209e2a02f34dd65d39"
format_stamp: "Formatted at 2026-06-21 02:49:33 on dist-test-slave-hvgn"
server_key: "5cc039d3458f247ace5d10337ffdee1e"
server_key_iv: "52a46e0d28df93d004e1643d485e7085"
server_key_version: "encryptionkey@0"
I20260621 02:49:33.936292 31135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "6eae4e68188343209e2a02f34dd65d39"
format_stamp: "Formatted at 2026-06-21 02:49:33 on dist-test-slave-hvgn"
server_key: "5cc039d3458f247ace5d10337ffdee1e"
server_key_iv: "52a46e0d28df93d004e1643d485e7085"
server_key_version: "encryptionkey@0"
I20260621 02:49:33.941218 31135 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:49:33.944545 31150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:33.945886 31135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:49:33.946079 31135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "6eae4e68188343209e2a02f34dd65d39"
format_stamp: "Formatted at 2026-06-21 02:49:33 on dist-test-slave-hvgn"
server_key: "5cc039d3458f247ace5d10337ffdee1e"
server_key_iv: "52a46e0d28df93d004e1643d485e7085"
server_key_version: "encryptionkey@0"
I20260621 02:49:33.946326 31135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:33.952090 31135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:33.956013 31135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:33.956303 31135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:33.974438 31135 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44047
I20260621 02:49:33.974529 31201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44047 every 8 connection(s)
I20260621 02:49:33.975965 31135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:33.979929 31202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:33.985020 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 31135
I20260621 02:49:33.985205 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:33.985625 9880 external_mini_cluster.cc:1428] Setting key 76ea13f96fa50e50e4773a1955d7c434
I20260621 02:49:33.993595 31202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39: Bootstrap starting.
I20260621 02:49:33.995694 31205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:33.987636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:44089 (local address 127.0.0.1:44047)
0621 02:49:33.988527 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:33.988544 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:33.988605 (+ 61us) server_negotiation.cc:407] Connection header received
0621 02:49:33.991469 (+ 2864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:33.991500 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:33.991977 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:33.992448 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:33.993557 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:33.993568 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:33.993594 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:33.993616 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:33.993693 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:33.993723 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:33.993965 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:33.994100 (+ 135us) server_negotiation.cc:299] Negotiation successful
0621 02:49:33.994317 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":342,"threads_started":1}
I20260621 02:49:33.997774 31202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:33.999166 31202 log.cc:824] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:33.999600 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:49:34.002525 31202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39: No bootstrap required, opened a new log
I20260621 02:49:34.008205 31205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:34.005116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:33947 (local address 127.0.0.1:44047)
0621 02:49:34.005334 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:34.005344 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:34.005784 (+ 440us) server_negotiation.cc:407] Connection header received
0621 02:49:34.005964 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:34.005967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:34.006020 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:34.006106 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:34.007261 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:34.007265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:34.007268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:34.007283 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:34.007321 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:34.007336 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:34.007602 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:34.007812 (+ 210us) server_negotiation.cc:299] Negotiation successful
0621 02:49:34.007901 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260621 02:49:34.012691 31202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eae4e68188343209e2a02f34dd65d39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44047 } }
I20260621 02:49:34.013135 31202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:34.013267 31202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eae4e68188343209e2a02f34dd65d39, State: Initialized, Role: FOLLOWER
I20260621 02:49:34.013904 31202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eae4e68188343209e2a02f34dd65d39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44047 } }
I20260621 02:49:34.014206 31202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:34.014384 31202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:34.014583 31202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:34.017089 31202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eae4e68188343209e2a02f34dd65d39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44047 } }
I20260621 02:49:34.017625 31202 leader_election.cc:302] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6eae4e68188343209e2a02f34dd65d39; no voters:
I20260621 02:49:34.018224 31202 leader_election.cc:288] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:34.018389 31212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:34.019131 31212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39 [term 1 LEADER]: Becoming Leader. State: Replica: 6eae4e68188343209e2a02f34dd65d39, State: Running, Role: LEADER
I20260621 02:49:34.019968 31212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eae4e68188343209e2a02f34dd65d39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44047 } }
I20260621 02:49:34.020658 31202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:34.023030 31214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6eae4e68188343209e2a02f34dd65d39. Latest consensus state: current_term: 1 leader_uuid: "6eae4e68188343209e2a02f34dd65d39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eae4e68188343209e2a02f34dd65d39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44047 } } }
I20260621 02:49:34.022977 31213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6eae4e68188343209e2a02f34dd65d39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6eae4e68188343209e2a02f34dd65d39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44047 } } }
I20260621 02:49:34.023957 31214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:34.024744 31213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:34.028682 31220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:34.033130 31220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:34.042009 31220 catalog_manager.cc:1347] Generated new cluster ID: cf780bdfc11e4780b2b3099dcb96731a
I20260621 02:49:34.042126 31220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:34.072477 31220 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:34.072968 31220 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:34.080181 31220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6eae4e68188343209e2a02f34dd65d39: Generated new TSK 0
I20260621 02:49:34.080947 31220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:34.090189 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 31135
W20260621 02:49:34.092118 31197 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.117:44089
W20260621 02:49:34.092867 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 31135: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:49:34.093025 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 31135
2026-06-21T02:49:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:34 dist-test-slave-hvgn krb5kdc[31233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:34 dist-test-slave-hvgn krb5kdc[31233](info): set up 2 sockets
Jun 21 02:49:34 dist-test-slave-hvgn krb5kdc[31233](info): commencing operation
krb5kdc: starting...
W20260621 02:49:35.207815 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:35 dist-test-slave-hvgn krb5kdc[31233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:49:35.452651 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40251
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:35.790666 31246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:35.790933 31246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:35.791021 31246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:35.798064 31246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:35.798170 31246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:35.798226 31246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:35.798286 31246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:35.798359 31246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:35.808261 31246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40251
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:35.809476 31246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:35.810672 31246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:35.822038 31251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:35.822090 31252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:35.822074 31254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:35.822892 31246 server_base.cc:1034] running on GCE node
I20260621 02:49:35.823663 31246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:35.825017 31246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:35.826252 31246 hybrid_clock.cc:648] HybridClock initialized: now 1782010175826188 us; error 50 us; skew 500 ppm
I20260621 02:49:35.826568 31246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:35 dist-test-slave-hvgn krb5kdc[31233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:35.829763 31246 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:49:35.831252 31246 webserver.cc:460] Webserver started at http://127.0.0.1:38327/ using document root <none> and password file <none>
I20260621 02:49:35.831948 31246 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:35.832119 31246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:35.832494 31246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:35.835755 31246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "ea1667625a49439286575f45f8c25cfe"
format_stamp: "Formatted at 2026-06-21 02:49:35 on dist-test-slave-hvgn"
server_key: "ae3d59bea0c79152d2791b1980455fa3"
server_key_iv: "cb0f18619d86c1a559ec77ca03fe9e71"
server_key_version: "encryptionkey@0"
I20260621 02:49:35.836457 31246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "ea1667625a49439286575f45f8c25cfe"
format_stamp: "Formatted at 2026-06-21 02:49:35 on dist-test-slave-hvgn"
server_key: "ae3d59bea0c79152d2791b1980455fa3"
server_key_iv: "cb0f18619d86c1a559ec77ca03fe9e71"
server_key_version: "encryptionkey@0"
I20260621 02:49:35.841389 31246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:49:35.844718 31261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:35.845985 31246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260621 02:49:35.846158 31246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "ea1667625a49439286575f45f8c25cfe"
format_stamp: "Formatted at 2026-06-21 02:49:35 on dist-test-slave-hvgn"
server_key: "ae3d59bea0c79152d2791b1980455fa3"
server_key_iv: "cb0f18619d86c1a559ec77ca03fe9e71"
server_key_version: "encryptionkey@0"
I20260621 02:49:35.846364 31246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:35.860430 31246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:35.864202 31246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:35.864478 31246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:35.882568 31246 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40251
I20260621 02:49:35.882596 31312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40251 every 8 connection(s)
I20260621 02:49:35.883900 31246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:35.885360 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 31246
I20260621 02:49:35.885547 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:35.886101 9880 external_mini_cluster.cc:1428] Setting key 841773948aedbb78f8533133aa6f7589
I20260621 02:49:35.888672 31313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 21 02:49:35 dist-test-slave-hvgn krb5kdc[31233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260621 02:49:35.901697 31313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe: Bootstrap starting.
I20260621 02:49:35.904850 31313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:35.905843 31313 log.cc:824] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:35.909030 31313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe: No bootstrap required, opened a new log
I20260621 02:49:35.909197 31316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:35.887907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:36913 (local address 127.0.0.1:40251)
0621 02:49:35.888824 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:35.888836 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:35.888879 (+ 43us) server_negotiation.cc:407] Connection header received
0621 02:49:35.890461 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:35.890481 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:35.890855 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:35.891188 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:35.892984 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:35.894361 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:35.896118 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:35.896749 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:35.896878 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:35.901649 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:35.901656 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:35.901671 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:35.901710 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:35.904705 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:35.905533 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:35.905539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:35.905545 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:35.905707 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:35.906241 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:35.906245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:35.906247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:35.907030 (+ 783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:35.907322 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:35.907877 (+ 555us) server_negotiation.cc:299] Negotiation successful
0621 02:49:35.908190 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":430,"threads_started":1}
I20260621 02:49:35.913075 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:49:35.917616 31313 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1667625a49439286575f45f8c25cfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40251 } }
I20260621 02:49:35.918143 31313 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:35.918273 31313 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea1667625a49439286575f45f8c25cfe, State: Initialized, Role: FOLLOWER
I20260621 02:49:35.918880 31313 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1667625a49439286575f45f8c25cfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40251 } }
I20260621 02:49:35.919122 31313 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:35.919263 31313 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:35.919469 31313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:35.921577 31313 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1667625a49439286575f45f8c25cfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40251 } }
I20260621 02:49:35.922045 31313 leader_election.cc:302] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea1667625a49439286575f45f8c25cfe; no voters:
I20260621 02:49:35.922606 31313 leader_election.cc:288] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:35.922756 31323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:35.923425 31323 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe [term 1 LEADER]: Becoming Leader. State: Replica: ea1667625a49439286575f45f8c25cfe, State: Running, Role: LEADER
I20260621 02:49:35.924036 31323 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1667625a49439286575f45f8c25cfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40251 } }
I20260621 02:49:35.925719 31313 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:35.927953 31324 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea1667625a49439286575f45f8c25cfe. Latest consensus state: current_term: 1 leader_uuid: "ea1667625a49439286575f45f8c25cfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1667625a49439286575f45f8c25cfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40251 } } }
I20260621 02:49:35.928329 31324 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:35.933260 31325 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea1667625a49439286575f45f8c25cfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1667625a49439286575f45f8c25cfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40251 } } }
I20260621 02:49:35.934121 31325 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:35.936538 31316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:35.917938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:58429 (local address 127.0.0.1:40251)
0621 02:49:35.918097 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:35.918105 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:35.918591 (+ 486us) server_negotiation.cc:407] Connection header received
0621 02:49:35.918778 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:35.918783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:35.918852 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:35.918958 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:35.920536 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:35.921300 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:35.922828 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:35.923835 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:35.924004 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:35.926946 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:35.926955 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:35.926959 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:35.926994 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:35.931449 (+ 4455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:35.932704 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:35.932712 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:35.932718 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:35.932978 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:35.933587 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:35.933591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:35.933594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:35.934083 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:35.934407 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:35.935765 (+ 1358us) server_negotiation.cc:299] Negotiation successful
0621 02:49:35.936006 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260621 02:49:35.938406 31328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:35.943905 31328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:35.952724 31328 catalog_manager.cc:1347] Generated new cluster ID: a0bd7812fb294f5aaab551e511831057
I20260621 02:49:35.952826 31328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:35.965965 31328 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:35.967163 31328 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:35.974577 31328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea1667625a49439286575f45f8c25cfe: Generated new TSK 0
I20260621 02:49:35.975642 31328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:35.985829 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 31246
I20260621 02:49:35.986999 31308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:36.075721 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 31246
W20260621 02:49:36.092674 31231 connection.cc:535] client connection to 127.0.0.1:40251 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40251): Connection reset by peer (error 104)
2026-06-21T02:49:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1982 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:36 dist-test-slave-hvgn krb5kdc[31345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:36 dist-test-slave-hvgn krb5kdc[31345](info): set up 2 sockets
Jun 21 02:49:36 dist-test-slave-hvgn krb5kdc[31345](info): commencing operation
krb5kdc: starting...
W20260621 02:49:37.175798 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.008s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:37 dist-test-slave-hvgn krb5kdc[31345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:49:37.422922 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33937
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:37.746858 31358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:37.747150 31358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:37.747320 31358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:37.756160 31358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:37.756278 31358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:37.756378 31358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:37.756479 31358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:37.756556 31358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:37.766428 31358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33937
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:37.767813 31358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:37.769132 31358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:37.780407 31363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:37.780366 31366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:37.780575 31364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:37.781617 31358 server_base.cc:1034] running on GCE node
I20260621 02:49:37.782380 31358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:37.783833 31358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:37.785055 31358 hybrid_clock.cc:648] HybridClock initialized: now 1782010177785009 us; error 28 us; skew 500 ppm
I20260621 02:49:37.785425 31358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:37 dist-test-slave-hvgn krb5kdc[31345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:37.788753 31358 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:49:37.790213 31358 webserver.cc:460] Webserver started at http://127.0.0.1:44065/ using document root <none> and password file <none>
I20260621 02:49:37.790875 31358 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:37.791005 31358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:37.791316 31358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:37.794371 31358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "666426385bce4eee9eb961cb392be781"
format_stamp: "Formatted at 2026-06-21 02:49:37 on dist-test-slave-hvgn"
server_key: "12fe35b6eedbe13909aa56f4c2c8e244"
server_key_iv: "68b513d67d5198a3a18445450377bf75"
server_key_version: "encryptionkey@0"
I20260621 02:49:37.795127 31358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "666426385bce4eee9eb961cb392be781"
format_stamp: "Formatted at 2026-06-21 02:49:37 on dist-test-slave-hvgn"
server_key: "12fe35b6eedbe13909aa56f4c2c8e244"
server_key_iv: "68b513d67d5198a3a18445450377bf75"
server_key_version: "encryptionkey@0"
I20260621 02:49:37.800202 31358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260621 02:49:37.803575 31373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:37.804857 31358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260621 02:49:37.805023 31358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "666426385bce4eee9eb961cb392be781"
format_stamp: "Formatted at 2026-06-21 02:49:37 on dist-test-slave-hvgn"
server_key: "12fe35b6eedbe13909aa56f4c2c8e244"
server_key_iv: "68b513d67d5198a3a18445450377bf75"
server_key_version: "encryptionkey@0"
I20260621 02:49:37.805199 31358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:37.826447 31358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:37.830401 31358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:37.830677 31358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:37.849774 31358 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33937
I20260621 02:49:37.849808 31424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33937 every 8 connection(s)
I20260621 02:49:37.851153 31358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:37.854779 31425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:37.855430 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 31358
I20260621 02:49:37.855618 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:37.856055 9880 external_mini_cluster.cc:1428] Setting key 38d41f9cc4f1cb1323807cdee8e2c86e
I20260621 02:49:37.869431 31425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781: Bootstrap starting.
I20260621 02:49:37.871616 31428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:37.858114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:37183 (local address 127.0.0.1:33937)
0621 02:49:37.858879 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:37.858898 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:37.858948 (+ 50us) server_negotiation.cc:407] Connection header received
0621 02:49:37.861603 (+ 2655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:37.861632 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:37.862166 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:37.862601 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:37.863099 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:37.864959 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:37.866317 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:37.867435 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:37.867599 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:37.868860 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:37.868869 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:37.868887 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:37.869592 (+ 705us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:37.869695 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:37.869728 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:37.869889 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:37.870103 (+ 214us) server_negotiation.cc:299] Negotiation successful
0621 02:49:37.870273 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":290,"threads_started":1}
I20260621 02:49:37.873109 31425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:37.874218 31425 log.cc:824] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:37.875546 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:49:37.877410 31425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781: No bootstrap required, opened a new log
I20260621 02:49:37.887591 31428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:37.880874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:50227 (local address 127.0.0.1:33937)
0621 02:49:37.881048 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:37.881055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:37.881417 (+ 362us) server_negotiation.cc:407] Connection header received
0621 02:49:37.881598 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:37.881604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:37.881684 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:37.881800 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:37.882265 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:37.883493 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:37.884842 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:37.885570 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:37.885740 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:37.886815 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:37.886821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:37.886834 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:37.886855 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:37.886896 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:37.886915 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:37.887032 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:37.887182 (+ 150us) server_negotiation.cc:299] Negotiation successful
0621 02:49:37.887233 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260621 02:49:37.887523 31425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666426385bce4eee9eb961cb392be781" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33937 } }
I20260621 02:49:37.888022 31425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:37.888150 31425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 666426385bce4eee9eb961cb392be781, State: Initialized, Role: FOLLOWER
I20260621 02:49:37.888881 31425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666426385bce4eee9eb961cb392be781" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33937 } }
I20260621 02:49:37.889181 31425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:37.889348 31425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:37.889484 31425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:37.891698 31425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666426385bce4eee9eb961cb392be781" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33937 } }
I20260621 02:49:37.892354 31425 leader_election.cc:302] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 666426385bce4eee9eb961cb392be781; no voters:
I20260621 02:49:37.892958 31425 leader_election.cc:288] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:37.893107 31435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:37.893864 31435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781 [term 1 LEADER]: Becoming Leader. State: Replica: 666426385bce4eee9eb961cb392be781, State: Running, Role: LEADER
I20260621 02:49:37.894425 31435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666426385bce4eee9eb961cb392be781" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33937 } }
I20260621 02:49:37.895032 31425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:37.897354 31437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 666426385bce4eee9eb961cb392be781. Latest consensus state: current_term: 1 leader_uuid: "666426385bce4eee9eb961cb392be781" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666426385bce4eee9eb961cb392be781" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33937 } } }
I20260621 02:49:37.897707 31437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:37.898898 31436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "666426385bce4eee9eb961cb392be781" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666426385bce4eee9eb961cb392be781" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33937 } } }
I20260621 02:49:37.899319 31436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:37.902885 31441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:37.907003 31441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:37.914814 31441 catalog_manager.cc:1347] Generated new cluster ID: 735b9d10d96242f580235ddfb4c9d6cc
I20260621 02:49:37.914921 31441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:37.951205 31441 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:37.953131 31441 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:37.963420 31441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 666426385bce4eee9eb961cb392be781: Generated new TSK 0
I20260621 02:49:37.964416 31441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:37.973667 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 31358
W20260621 02:49:37.975253 31420 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.117:37183
W20260621 02:49:37.976011 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 31358: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:49:37.976152 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 31358
W20260621 02:49:37.992856 31343 connection.cc:535] client connection to 127.0.0.1:33937 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33937): Connection reset by peer (error 104)
2026-06-21T02:49:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1900 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:38 dist-test-slave-hvgn krb5kdc[31456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:38 dist-test-slave-hvgn krb5kdc[31456](info): set up 2 sockets
Jun 21 02:49:38 dist-test-slave-hvgn krb5kdc[31456](info): commencing operation
krb5kdc: starting...
W20260621 02:49:39.067752 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:39 dist-test-slave-hvgn krb5kdc[31456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260621 02:49:39.328259 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35003
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:39.660012 31469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:39.660288 31469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:39.660436 31469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:39.667594 31469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:39.667699 31469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:39.667820 31469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:39.667917 31469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:39.667994 31469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:39.677778 31469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35003
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:39.679157 31469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:39.680482 31469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:39.691871 31475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:39.692041 31474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:39.691833 31477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:39.692798 31469 server_base.cc:1034] running on GCE node
I20260621 02:49:39.693511 31469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:39.694867 31469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:39.696087 31469 hybrid_clock.cc:648] HybridClock initialized: now 1782010179696043 us; error 56 us; skew 500 ppm
I20260621 02:49:39.696424 31469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:39 dist-test-slave-hvgn krb5kdc[31456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:39.699826 31469 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260621 02:49:39.701259 31469 webserver.cc:460] Webserver started at http://127.0.0.1:39735/ using document root <none> and password file <none>
I20260621 02:49:39.701864 31469 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:39.701960 31469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:39.702195 31469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:39.705090 31469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "6d6b2d09904b45548fd3f766449505cc"
format_stamp: "Formatted at 2026-06-21 02:49:39 on dist-test-slave-hvgn"
server_key: "0a008095a808eac7b438c6b0bf406bbc"
server_key_iv: "683ecaac68cb75becba3ad9d7984b926"
server_key_version: "encryptionkey@0"
I20260621 02:49:39.705799 31469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "6d6b2d09904b45548fd3f766449505cc"
format_stamp: "Formatted at 2026-06-21 02:49:39 on dist-test-slave-hvgn"
server_key: "0a008095a808eac7b438c6b0bf406bbc"
server_key_iv: "683ecaac68cb75becba3ad9d7984b926"
server_key_version: "encryptionkey@0"
I20260621 02:49:39.710716 31469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260621 02:49:39.714039 31484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:39.715456 31469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260621 02:49:39.715601 31469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "6d6b2d09904b45548fd3f766449505cc"
format_stamp: "Formatted at 2026-06-21 02:49:39 on dist-test-slave-hvgn"
server_key: "0a008095a808eac7b438c6b0bf406bbc"
server_key_iv: "683ecaac68cb75becba3ad9d7984b926"
server_key_version: "encryptionkey@0"
I20260621 02:49:39.715761 31469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:39.721246 31469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:39.725407 31469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:39.725642 31469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:39.743989 31469 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35003
I20260621 02:49:39.744035 31535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35003 every 8 connection(s)
I20260621 02:49:39.745440 31469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:39.749238 31536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:39.750237 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 31469
I20260621 02:49:39.750415 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:39.750885 9880 external_mini_cluster.cc:1428] Setting key 202aaabf8222c0ed9e12ec9a956a4196
I20260621 02:49:39.759441 31539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:39.752775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:58971 (local address 127.0.0.1:35003)
0621 02:49:39.753595 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:39.753606 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:39.753646 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:49:39.755225 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:39.755246 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:39.755649 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:39.756075 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:39.757361 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:39.757370 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:39.757393 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:39.757408 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:39.757521 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:39.757558 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:39.757835 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:39.757988 (+ 153us) server_negotiation.cc:299] Negotiation successful
0621 02:49:39.758181 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":370,"threads_started":1}
I20260621 02:49:39.763144 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:49:39.763252 31536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc: Bootstrap starting.
I20260621 02:49:39.766281 31536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:39.767436 31536 log.cc:824] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:39.770203 31539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:39.767656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.117:54525 (local address 127.0.0.1:35003)
0621 02:49:39.767900 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:39.767908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:39.768317 (+ 409us) server_negotiation.cc:407] Connection header received
0621 02:49:39.768454 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:39.768457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:39.768502 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:39.768590 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:39.769564 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:39.769570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:39.769574 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:39.769592 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:39.769641 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:39.769656 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:39.769779 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:39.769883 (+ 104us) server_negotiation.cc:299] Negotiation successful
0621 02:49:39.769943 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260621 02:49:39.770572 31536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc: No bootstrap required, opened a new log
I20260621 02:49:39.777864 31536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6b2d09904b45548fd3f766449505cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35003 } }
I20260621 02:49:39.778308 31536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:39.778437 31536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d6b2d09904b45548fd3f766449505cc, State: Initialized, Role: FOLLOWER
I20260621 02:49:39.779027 31536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6b2d09904b45548fd3f766449505cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35003 } }
I20260621 02:49:39.779227 31536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:39.779336 31536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:39.779513 31536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:39.781534 31536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6b2d09904b45548fd3f766449505cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35003 } }
I20260621 02:49:39.781951 31536 leader_election.cc:302] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d6b2d09904b45548fd3f766449505cc; no voters:
I20260621 02:49:39.782431 31536 leader_election.cc:288] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:39.782582 31546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:39.783231 31546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc [term 1 LEADER]: Becoming Leader. State: Replica: 6d6b2d09904b45548fd3f766449505cc, State: Running, Role: LEADER
I20260621 02:49:39.783784 31546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6b2d09904b45548fd3f766449505cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35003 } }
I20260621 02:49:39.784464 31536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:39.787112 31547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d6b2d09904b45548fd3f766449505cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6b2d09904b45548fd3f766449505cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35003 } } }
I20260621 02:49:39.787464 31547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:39.788244 31548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d6b2d09904b45548fd3f766449505cc. Latest consensus state: current_term: 1 leader_uuid: "6d6b2d09904b45548fd3f766449505cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6b2d09904b45548fd3f766449505cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35003 } } }
I20260621 02:49:39.788591 31548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:39.788975 31551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:39.794193 31551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:39.802521 31551 catalog_manager.cc:1347] Generated new cluster ID: fa3ebf90fd5c457e9062b895abd3695f
I20260621 02:49:39.802625 31551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:39.822856 31551 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:39.823271 31551 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:39.829443 31551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d6b2d09904b45548fd3f766449505cc: Generated new TSK 0
I20260621 02:49:39.830344 31551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:39.842801 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 31469
W20260621 02:49:39.843904 31531 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.117:58971
W20260621 02:49:39.844596 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 31469: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:49:39.844733 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 31469
2026-06-21T02:49:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1868 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:39 dist-test-slave-hvgn krb5kdc[31567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:39 dist-test-slave-hvgn krb5kdc[31567](info): set up 2 sockets
Jun 21 02:49:39 dist-test-slave-hvgn krb5kdc[31567](info): commencing operation
krb5kdc: starting...
W20260621 02:49:40.939760 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:41 dist-test-slave-hvgn krb5kdc[31567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:49:41.182155 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:40075
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:40075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:41.507642 31580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:41.508040 31580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:41.508203 31580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:41.515635 31580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:41.515770 31580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:41.515869 31580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:41.515969 31580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:41.516048 31580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:41.525624 31580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:40075
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:40075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:41.526924 31580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:41.528280 31580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:41.540387 31586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:41.540997 31588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:41.541028 31585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:41.541525 31580 server_base.cc:1034] running on GCE node
I20260621 02:49:41.542347 31580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:41.543798 31580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:41.545032 31580 hybrid_clock.cc:648] HybridClock initialized: now 1782010181544971 us; error 53 us; skew 500 ppm
I20260621 02:49:41.545388 31580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:41 dist-test-slave-hvgn krb5kdc[31567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010181, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:41.548565 31580 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:49:41.550133 31580 webserver.cc:460] Webserver started at http://127.9.166.62:35457/ using document root <none> and password file <none>
I20260621 02:49:41.550859 31580 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:41.551007 31580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:41.551313 31580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:41.554342 31580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "ccf1258bb9604dd9a79f53df550fc00d"
format_stamp: "Formatted at 2026-06-21 02:49:41 on dist-test-slave-hvgn"
server_key: "d268efd9ea098c7c7638a757677ce0b7"
server_key_iv: "7f6168510ccffe126861c324e9d78c08"
server_key_version: "encryptionkey@0"
I20260621 02:49:41.555100 31580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "ccf1258bb9604dd9a79f53df550fc00d"
format_stamp: "Formatted at 2026-06-21 02:49:41 on dist-test-slave-hvgn"
server_key: "d268efd9ea098c7c7638a757677ce0b7"
server_key_iv: "7f6168510ccffe126861c324e9d78c08"
server_key_version: "encryptionkey@0"
I20260621 02:49:41.560444 31580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260621 02:49:41.563917 31595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:41.565292 31580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260621 02:49:41.565521 31580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "ccf1258bb9604dd9a79f53df550fc00d"
format_stamp: "Formatted at 2026-06-21 02:49:41 on dist-test-slave-hvgn"
server_key: "d268efd9ea098c7c7638a757677ce0b7"
server_key_iv: "7f6168510ccffe126861c324e9d78c08"
server_key_version: "encryptionkey@0"
I20260621 02:49:41.565721 31580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:41.581697 31580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:41.585948 31580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:41.586249 31580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:41.604189 31580 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:40075
I20260621 02:49:41.604249 31646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:40075 every 8 connection(s)
I20260621 02:49:41.605520 31580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:41.609180 31647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:41.610306 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 31580
I20260621 02:49:41.610541 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:41.610996 9880 external_mini_cluster.cc:1428] Setting key f842c5f3c023a6565c128d7d4d56ca9d
I20260621 02:49:41.623060 31647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d: Bootstrap starting.
Jun 21 02:49:41 dist-test-slave-hvgn krb5kdc[31567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:49:41.626226 31647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:41.627604 31647 log.cc:824] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:41.630955 31647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d: No bootstrap required, opened a new log
I20260621 02:49:41.635210 31650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:41.612815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56891 (local address 127.9.166.62:40075)
0621 02:49:41.613629 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:41.613644 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:41.613693 (+ 49us) server_negotiation.cc:407] Connection header received
0621 02:49:41.615596 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:41.615623 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:41.616119 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:41.616555 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:41.617801 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:41.619800 (+ 1999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:41.621033 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:41.622077 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:41.622271 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:41.626359 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:41.626369 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:41.626392 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:41.626439 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:41.630497 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:41.631295 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:41.631303 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:41.631311 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:41.631558 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:41.631992 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:41.631997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:41.632000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:41.632705 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:41.633049 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:41.633424 (+ 375us) server_negotiation.cc:299] Negotiation successful
0621 02:49:41.633762 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":350,"threads_started":1}
I20260621 02:49:41.639222 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:49:41.640695 31647 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf1258bb9604dd9a79f53df550fc00d" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40075 } }
I20260621 02:49:41.641044 31647 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:41.641137 31647 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf1258bb9604dd9a79f53df550fc00d, State: Initialized, Role: FOLLOWER
I20260621 02:49:41.641690 31647 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf1258bb9604dd9a79f53df550fc00d" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40075 } }
I20260621 02:49:41.641937 31647 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:41.642091 31647 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:41.642258 31647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:41.645399 31647 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf1258bb9604dd9a79f53df550fc00d" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40075 } }
I20260621 02:49:41.646024 31647 leader_election.cc:302] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccf1258bb9604dd9a79f53df550fc00d; no voters:
I20260621 02:49:41.646750 31647 leader_election.cc:288] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:41.647497 31657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:41.648219 31657 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d [term 1 LEADER]: Becoming Leader. State: Replica: ccf1258bb9604dd9a79f53df550fc00d, State: Running, Role: LEADER
I20260621 02:49:41.648787 31657 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf1258bb9604dd9a79f53df550fc00d" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40075 } }
I20260621 02:49:41.649004 31647 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:41.651721 31658 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccf1258bb9604dd9a79f53df550fc00d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf1258bb9604dd9a79f53df550fc00d" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40075 } } }
I20260621 02:49:41.653342 31658 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:41.652736 31659 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccf1258bb9604dd9a79f53df550fc00d. Latest consensus state: current_term: 1 leader_uuid: "ccf1258bb9604dd9a79f53df550fc00d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf1258bb9604dd9a79f53df550fc00d" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40075 } } }
I20260621 02:49:41.653738 31659 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:41.660915 31664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:41.661192 31650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:41.643572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35317 (local address 127.9.166.62:40075)
0621 02:49:41.643773 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:41.643780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:41.644332 (+ 552us) server_negotiation.cc:407] Connection header received
0621 02:49:41.644476 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:41.644480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:41.644537 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:41.644614 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:41.646687 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:41.647768 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:41.649502 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:41.650143 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:41.650289 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:41.653751 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:41.653762 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:41.653767 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:41.653805 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:41.658147 (+ 4342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:41.658852 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:41.658859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:41.658862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:41.658999 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:41.659501 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:41.659508 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:41.659512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:41.659818 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:41.660100 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:41.660481 (+ 381us) server_negotiation.cc:299] Negotiation successful
0621 02:49:41.660844 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:49:41.666127 31664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:41.676563 31664 catalog_manager.cc:1347] Generated new cluster ID: eb37cd2924ca48849b6ac568af267471
I20260621 02:49:41.676702 31664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:41.689239 31664 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:41.690548 31664 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:41.703017 31664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccf1258bb9604dd9a79f53df550fc00d: Generated new TSK 0
I20260621 02:49:41.703912 31664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:41.717579 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 31580
I20260621 02:49:41.718710 31642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:41.810947 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 31580
W20260621 02:49:41.828157 31565 connection.cc:535] client connection to 127.9.166.62:40075 recv error: Network error: failed to read from TLS socket (remote: 127.9.166.62:40075): Connection reset by peer (error 104)
2026-06-21T02:49:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1966 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:41 dist-test-slave-hvgn krb5kdc[31679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:41 dist-test-slave-hvgn krb5kdc[31679](info): set up 2 sockets
Jun 21 02:49:41 dist-test-slave-hvgn krb5kdc[31679](info): commencing operation
krb5kdc: starting...
W20260621 02:49:42.907943 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:43 dist-test-slave-hvgn krb5kdc[31679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:49:43.154227 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:41801
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:45319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:41801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:43.479686 31692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:43.480014 31692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:43.480180 31692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:43.487239 31692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:43.487430 31692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:43.487540 31692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:43.487648 31692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:43.487730 31692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:43.497344 31692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:45319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:41801
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:41801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:43.498600 31692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:43.499930 31692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:43.511528 31697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:43.511567 31700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:43.511504 31698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:43.512305 31692 server_base.cc:1034] running on GCE node
I20260621 02:49:43.513093 31692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:43.514616 31692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:43.515839 31692 hybrid_clock.cc:648] HybridClock initialized: now 1782010183515807 us; error 57 us; skew 500 ppm
I20260621 02:49:43.516183 31692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:43 dist-test-slave-hvgn krb5kdc[31679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010183, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:43.519326 31692 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:49:43.521217 31692 webserver.cc:460] Webserver started at http://127.9.166.62:36581/ using document root <none> and password file <none>
I20260621 02:49:43.521852 31692 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:43.521951 31692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:43.522207 31692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:43.525137 31692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "10c50fe7ea7c4a53bbe618cf4a36dc82"
format_stamp: "Formatted at 2026-06-21 02:49:43 on dist-test-slave-hvgn"
server_key: "8ac86e71c71303125b0d8463a10702b9"
server_key_iv: "4c06bd5e1e7c45169f0d14f4de50f54d"
server_key_version: "encryptionkey@0"
I20260621 02:49:43.525852 31692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "10c50fe7ea7c4a53bbe618cf4a36dc82"
format_stamp: "Formatted at 2026-06-21 02:49:43 on dist-test-slave-hvgn"
server_key: "8ac86e71c71303125b0d8463a10702b9"
server_key_iv: "4c06bd5e1e7c45169f0d14f4de50f54d"
server_key_version: "encryptionkey@0"
I20260621 02:49:43.530905 31692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 02:49:43.534463 31707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:43.536013 31692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:49:43.536242 31692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "10c50fe7ea7c4a53bbe618cf4a36dc82"
format_stamp: "Formatted at 2026-06-21 02:49:43 on dist-test-slave-hvgn"
server_key: "8ac86e71c71303125b0d8463a10702b9"
server_key_iv: "4c06bd5e1e7c45169f0d14f4de50f54d"
server_key_version: "encryptionkey@0"
I20260621 02:49:43.536441 31692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:43.551941 31692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:43.555927 31692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:43.556193 31692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:43.574615 31692 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:41801
I20260621 02:49:43.574657 31758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:41801 every 8 connection(s)
I20260621 02:49:43.576045 31692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:43.577306 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 31692
I20260621 02:49:43.577489 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:43.578070 9880 external_mini_cluster.cc:1428] Setting key a0e2445bed3929387127ae498b2d2893
I20260621 02:49:43.580739 31759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:43.593217 31762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:43.579834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55347 (local address 127.9.166.62:41801)
0621 02:49:43.580604 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:43.580618 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:43.580667 (+ 49us) server_negotiation.cc:407] Connection header received
0621 02:49:43.583047 (+ 2380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:43.583074 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:43.583570 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:43.584032 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:43.584537 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:43.586683 (+ 2146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:43.588066 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:43.589056 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:43.589256 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:43.590368 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:43.590379 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:43.590397 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:43.591165 (+ 768us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:43.591262 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:43.591298 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:43.591475 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:43.591774 (+ 299us) server_negotiation.cc:299] Negotiation successful
0621 02:49:43.591950 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":331,"threads_started":1}
I20260621 02:49:43.594836 31759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82: Bootstrap starting.
I20260621 02:49:43.596802 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:49:43.598296 31759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:43.599521 31759 log.cc:824] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:43.602253 31759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82: No bootstrap required, opened a new log
I20260621 02:49:43.609150 31762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:43.601943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37561 (local address 127.9.166.62:41801)
0621 02:49:43.602134 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:43.602142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:43.602669 (+ 527us) server_negotiation.cc:407] Connection header received
0621 02:49:43.602877 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:43.602881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:43.602935 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:43.603014 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:43.603634 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:43.604400 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:43.606312 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:43.607003 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:43.607114 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:43.608317 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:43.608324 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:43.608329 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:43.608354 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:43.608400 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:43.608419 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:43.608536 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:43.608756 (+ 220us) server_negotiation.cc:299] Negotiation successful
0621 02:49:43.608816 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:49:43.611920 31759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c50fe7ea7c4a53bbe618cf4a36dc82" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41801 } }
I20260621 02:49:43.612274 31759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:43.612394 31759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c50fe7ea7c4a53bbe618cf4a36dc82, State: Initialized, Role: FOLLOWER
I20260621 02:49:43.612968 31759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c50fe7ea7c4a53bbe618cf4a36dc82" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41801 } }
I20260621 02:49:43.613196 31759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:43.613346 31759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:43.613515 31759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:43.615653 31759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c50fe7ea7c4a53bbe618cf4a36dc82" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41801 } }
I20260621 02:49:43.616134 31759 leader_election.cc:302] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10c50fe7ea7c4a53bbe618cf4a36dc82; no voters:
I20260621 02:49:43.616792 31759 leader_election.cc:288] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:43.616897 31769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:43.617704 31769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82 [term 1 LEADER]: Becoming Leader. State: Replica: 10c50fe7ea7c4a53bbe618cf4a36dc82, State: Running, Role: LEADER
I20260621 02:49:43.618220 31769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c50fe7ea7c4a53bbe618cf4a36dc82" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41801 } }
I20260621 02:49:43.618917 31759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:43.621191 31770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10c50fe7ea7c4a53bbe618cf4a36dc82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c50fe7ea7c4a53bbe618cf4a36dc82" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41801 } } }
I20260621 02:49:43.621641 31770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:43.621229 31771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10c50fe7ea7c4a53bbe618cf4a36dc82. Latest consensus state: current_term: 1 leader_uuid: "10c50fe7ea7c4a53bbe618cf4a36dc82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c50fe7ea7c4a53bbe618cf4a36dc82" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41801 } } }
I20260621 02:49:43.622570 31771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:43.623270 31777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:43.628698 31777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:43.629344 31786 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 02:49:43.637123 31777 catalog_manager.cc:1347] Generated new cluster ID: 1c2f3278d009466eb5dc62ddbe2b0290
I20260621 02:49:43.637217 31777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:43.653662 31777 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:43.655154 31777 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:43.661661 31777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10c50fe7ea7c4a53bbe618cf4a36dc82: Generated new TSK 0
I20260621 02:49:43.662487 31777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:43.670900 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 31692
W20260621 02:49:43.672436 31754 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55347
W20260621 02:49:43.673224 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 31692: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:49:43.673393 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 31692
W20260621 02:49:43.690049 31677 connection.cc:535] client connection to 127.9.166.62:41801 recv error: Network error: failed to read from TLS socket (remote: 127.9.166.62:41801): Connection reset by peer (error 104)
2026-06-21T02:49:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1861 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:43 dist-test-slave-hvgn krb5kdc[31790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:43 dist-test-slave-hvgn krb5kdc[31790](info): set up 2 sockets
Jun 21 02:49:43 dist-test-slave-hvgn krb5kdc[31790](info): commencing operation
krb5kdc: starting...
W20260621 02:49:44.763903 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:44 dist-test-slave-hvgn krb5kdc[31790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:49:45.010321 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:40487
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:40487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:45.333483 31803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:45.333770 31803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:45.333861 31803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:45.340965 31803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:45.341076 31803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:45.341123 31803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:45.341185 31803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:45.341233 31803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:45.350858 31803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:40487
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:40487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:45.352063 31803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:45.353289 31803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:45.364918 31808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:45.364988 31811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:45.365615 31809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:45.365694 31803 server_base.cc:1034] running on GCE node
I20260621 02:49:45.366516 31803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:45.367961 31803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:45.369196 31803 hybrid_clock.cc:648] HybridClock initialized: now 1782010185369155 us; error 50 us; skew 500 ppm
I20260621 02:49:45.369572 31803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:45 dist-test-slave-hvgn krb5kdc[31790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010185, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:45.376097 31803 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:49:45.377548 31803 webserver.cc:460] Webserver started at http://127.9.166.62:33351/ using document root <none> and password file <none>
I20260621 02:49:45.378477 31803 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:45.378661 31803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:45.379022 31803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:45.382547 31803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "ebdc99ce1a244ce297d83c9507fa41e1"
format_stamp: "Formatted at 2026-06-21 02:49:45 on dist-test-slave-hvgn"
server_key: "a3786bf8fe74f54ddf98bc66b091cb33"
server_key_iv: "2f4ffd13d2ad89ac5add157d84f3d902"
server_key_version: "encryptionkey@0"
I20260621 02:49:45.383468 31803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "ebdc99ce1a244ce297d83c9507fa41e1"
format_stamp: "Formatted at 2026-06-21 02:49:45 on dist-test-slave-hvgn"
server_key: "a3786bf8fe74f54ddf98bc66b091cb33"
server_key_iv: "2f4ffd13d2ad89ac5add157d84f3d902"
server_key_version: "encryptionkey@0"
I20260621 02:49:45.388546 31803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260621 02:49:45.391950 31818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:45.393257 31803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260621 02:49:45.393455 31803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "ebdc99ce1a244ce297d83c9507fa41e1"
format_stamp: "Formatted at 2026-06-21 02:49:45 on dist-test-slave-hvgn"
server_key: "a3786bf8fe74f54ddf98bc66b091cb33"
server_key_iv: "2f4ffd13d2ad89ac5add157d84f3d902"
server_key_version: "encryptionkey@0"
I20260621 02:49:45.393640 31803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:45.399088 31803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:45.402858 31803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:45.403146 31803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:45.421175 31803 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:40487
I20260621 02:49:45.421223 31869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:40487 every 8 connection(s)
I20260621 02:49:45.422539 31803 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:45.422818 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 31803
I20260621 02:49:45.422972 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:45.423473 9880 external_mini_cluster.cc:1428] Setting key 895241d2d45edf67f5b2964c9abbe119
I20260621 02:49:45.426895 31870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:45.431023 31873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:45.425323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45859 (local address 127.9.166.62:40487)
0621 02:49:45.426066 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:45.426078 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:45.426116 (+ 38us) server_negotiation.cc:407] Connection header received
0621 02:49:45.427794 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:45.427815 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:45.428118 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:45.428472 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:45.429468 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:45.429473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:45.429488 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:45.429506 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:45.429573 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:45.429594 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:45.429857 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:45.429991 (+ 134us) server_negotiation.cc:299] Negotiation successful
0621 02:49:45.430170 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":327,"threads_started":1}
I20260621 02:49:45.434708 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:49:45.438220 31870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1: Bootstrap starting.
I20260621 02:49:45.441394 31873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:45.439033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60575 (local address 127.9.166.62:40487)
0621 02:49:45.439203 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:45.439208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:45.439651 (+ 443us) server_negotiation.cc:407] Connection header received
0621 02:49:45.439772 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:45.439777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:45.439845 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:45.439948 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:45.440796 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:45.440802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:45.440809 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:45.440830 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:45.440873 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:45.440892 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:45.441016 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:45.441070 (+ 54us) server_negotiation.cc:299] Negotiation successful
0621 02:49:45.441125 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:49:45.441908 31870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:45.443074 31870 log.cc:824] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:45.445533 31870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1: No bootstrap required, opened a new log
I20260621 02:49:45.452641 31870 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdc99ce1a244ce297d83c9507fa41e1" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40487 } }
I20260621 02:49:45.452960 31870 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:45.453030 31870 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebdc99ce1a244ce297d83c9507fa41e1, State: Initialized, Role: FOLLOWER
I20260621 02:49:45.453480 31870 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdc99ce1a244ce297d83c9507fa41e1" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40487 } }
I20260621 02:49:45.453663 31870 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:45.453758 31870 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:45.453883 31870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:45.455966 31870 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdc99ce1a244ce297d83c9507fa41e1" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40487 } }
I20260621 02:49:45.456440 31870 leader_election.cc:302] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebdc99ce1a244ce297d83c9507fa41e1; no voters:
I20260621 02:49:45.457000 31870 leader_election.cc:288] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:45.457130 31880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:45.458504 31880 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1 [term 1 LEADER]: Becoming Leader. State: Replica: ebdc99ce1a244ce297d83c9507fa41e1, State: Running, Role: LEADER
I20260621 02:49:45.458999 31880 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdc99ce1a244ce297d83c9507fa41e1" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40487 } }
I20260621 02:49:45.459115 31870 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:45.461884 31881 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebdc99ce1a244ce297d83c9507fa41e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdc99ce1a244ce297d83c9507fa41e1" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40487 } } }
I20260621 02:49:45.462280 31881 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:45.463042 31882 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebdc99ce1a244ce297d83c9507fa41e1. Latest consensus state: current_term: 1 leader_uuid: "ebdc99ce1a244ce297d83c9507fa41e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdc99ce1a244ce297d83c9507fa41e1" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40487 } } }
I20260621 02:49:45.463620 31882 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:45.465385 31888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:45.470618 31888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:45.479233 31888 catalog_manager.cc:1347] Generated new cluster ID: 339c3f7d5a90466fa014e22815c8c945
I20260621 02:49:45.479373 31888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:45.500934 31888 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:45.501334 31888 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:45.513427 31888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebdc99ce1a244ce297d83c9507fa41e1: Generated new TSK 0
I20260621 02:49:45.514356 31888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:45.530073 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 31803
W20260621 02:49:45.531059 31865 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45859
W20260621 02:49:45.531868 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 31803: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:49:45.532011 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 31803
2026-06-21T02:49:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1858 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:45 dist-test-slave-hvgn krb5kdc[31901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:45 dist-test-slave-hvgn krb5kdc[31901](info): set up 2 sockets
Jun 21 02:49:45 dist-test-slave-hvgn krb5kdc[31901](info): commencing operation
krb5kdc: starting...
W20260621 02:49:46.667945 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:46 dist-test-slave-hvgn krb5kdc[31901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:49:46.918748 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:40633
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:38817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:40633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:47.245029 31914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:47.245391 31914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:47.245565 31914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:47.252732 31914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:47.252880 31914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:47.252982 31914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:47.253094 31914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:47.253175 31914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:47.262907 31914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:38817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:40633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:40633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:47.264230 31914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:47.266304 31914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:47.278764 31919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:47.278954 31922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:47.279212 31920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:47.279696 31914 server_base.cc:1034] running on GCE node
I20260621 02:49:47.280479 31914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:47.281800 31914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:47.283036 31914 hybrid_clock.cc:648] HybridClock initialized: now 1782010187283005 us; error 53 us; skew 500 ppm
I20260621 02:49:47.283432 31914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:47 dist-test-slave-hvgn krb5kdc[31901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010187, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:47.286497 31914 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:49:47.288034 31914 webserver.cc:460] Webserver started at http://127.9.166.62:36313/ using document root <none> and password file <none>
I20260621 02:49:47.288712 31914 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:47.288857 31914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:47.289149 31914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:47.292081 31914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "809bc482b38640958200a35ca1fd5bb6"
format_stamp: "Formatted at 2026-06-21 02:49:47 on dist-test-slave-hvgn"
server_key: "b16191ce51ca1573615e19848163cd13"
server_key_iv: "d1794558b7f4848ce1a4a3886c6fdd9e"
server_key_version: "encryptionkey@0"
I20260621 02:49:47.292848 31914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "809bc482b38640958200a35ca1fd5bb6"
format_stamp: "Formatted at 2026-06-21 02:49:47 on dist-test-slave-hvgn"
server_key: "b16191ce51ca1573615e19848163cd13"
server_key_iv: "d1794558b7f4848ce1a4a3886c6fdd9e"
server_key_version: "encryptionkey@0"
I20260621 02:49:47.297806 31914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260621 02:49:47.301131 31929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:47.302480 31914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:49:47.302688 31914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "809bc482b38640958200a35ca1fd5bb6"
format_stamp: "Formatted at 2026-06-21 02:49:47 on dist-test-slave-hvgn"
server_key: "b16191ce51ca1573615e19848163cd13"
server_key_iv: "d1794558b7f4848ce1a4a3886c6fdd9e"
server_key_version: "encryptionkey@0"
I20260621 02:49:47.302887 31914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:47.330956 31914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:47.335431 31914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:47.335722 31914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:47.354136 31914 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:40633
I20260621 02:49:47.354341 31980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:40633 every 8 connection(s)
I20260621 02:49:47.355556 31914 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:47.359249 31981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:47.361835 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 31914
I20260621 02:49:47.362012 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:47.362542 9880 external_mini_cluster.cc:1428] Setting key 9b4bbbe47be03f594b7433aeab49e739
I20260621 02:49:47.372014 31981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6: Bootstrap starting.
I20260621 02:49:47.375964 31981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:47.377162 31981 log.cc:824] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6: Log is configured to *not* fsync() on all Append() calls
Jun 21 02:49:47 dist-test-slave-hvgn krb5kdc[31901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:49:47.380793 31981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6: No bootstrap required, opened a new log
I20260621 02:49:47.387180 31984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:47.364375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41429 (local address 127.9.166.62:40633)
0621 02:49:47.365205 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:47.365217 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:47.365266 (+ 49us) server_negotiation.cc:407] Connection header received
0621 02:49:47.366961 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:47.366984 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:47.367372 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:47.367775 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:47.369554 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:47.370861 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:47.372674 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:47.373317 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:47.373450 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:47.378604 (+ 5154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:47.378616 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:47.378634 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:47.378680 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:47.382631 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:47.383321 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:47.383334 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:47.383342 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:47.383563 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:47.383938 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:47.383943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:47.383946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:47.384679 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:47.384976 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:47.385351 (+ 375us) server_negotiation.cc:299] Negotiation successful
0621 02:49:47.385772 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":382,"threads_started":1}
I20260621 02:49:47.390960 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:49:47.391148 31981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809bc482b38640958200a35ca1fd5bb6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40633 } }
I20260621 02:49:47.391512 31981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:47.391614 31981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809bc482b38640958200a35ca1fd5bb6, State: Initialized, Role: FOLLOWER
I20260621 02:49:47.392132 31981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809bc482b38640958200a35ca1fd5bb6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40633 } }
I20260621 02:49:47.392377 31981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:47.392519 31981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:47.392686 31981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:47.394752 31981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809bc482b38640958200a35ca1fd5bb6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40633 } }
I20260621 02:49:47.395219 31981 leader_election.cc:302] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 809bc482b38640958200a35ca1fd5bb6; no voters:
I20260621 02:49:47.395989 31981 leader_election.cc:288] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:47.396101 31990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:47.396763 31990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6 [term 1 LEADER]: Becoming Leader. State: Replica: 809bc482b38640958200a35ca1fd5bb6, State: Running, Role: LEADER
I20260621 02:49:47.397512 31990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809bc482b38640958200a35ca1fd5bb6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40633 } }
I20260621 02:49:47.399694 31992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 809bc482b38640958200a35ca1fd5bb6. Latest consensus state: current_term: 1 leader_uuid: "809bc482b38640958200a35ca1fd5bb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809bc482b38640958200a35ca1fd5bb6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40633 } } }
I20260621 02:49:47.400127 31992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:47.400848 31981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:47.403561 31993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "809bc482b38640958200a35ca1fd5bb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809bc482b38640958200a35ca1fd5bb6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 40633 } } }
I20260621 02:49:47.403961 31993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:47.408092 31995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:47.414636 31984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:47.395820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39507 (local address 127.9.166.62:40633)
0621 02:49:47.396046 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:47.396054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:47.396620 (+ 566us) server_negotiation.cc:407] Connection header received
0621 02:49:47.396919 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:47.396925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:47.396998 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:47.397060 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:47.398729 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:47.399950 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:47.401734 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:47.402779 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:47.402939 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:47.406953 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:47.406960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:47.406964 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:47.406998 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:47.410643 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:47.411529 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:47.411534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:47.411537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:47.411683 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:47.412161 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:47.412168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:47.412173 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:47.412588 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:47.412838 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:47.414021 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0621 02:49:47.414250 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":64}
I20260621 02:49:47.414678 31995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:47.425004 31995 catalog_manager.cc:1347] Generated new cluster ID: 2fa970bf2d704fa3a45021f22601ed2b
I20260621 02:49:47.425132 31995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:47.444062 31995 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:47.446007 31995 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:47.457137 31995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 809bc482b38640958200a35ca1fd5bb6: Generated new TSK 0
I20260621 02:49:47.457989 31995 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:47.467363 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 31914
I20260621 02:49:47.468430 31976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:47.558709 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 31914
2026-06-21T02:49:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2024 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:47 dist-test-slave-hvgn krb5kdc[32013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:47 dist-test-slave-hvgn krb5kdc[32013](info): set up 2 sockets
Jun 21 02:49:47 dist-test-slave-hvgn krb5kdc[32013](info): commencing operation
krb5kdc: starting...
W20260621 02:49:48.647975 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.008s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:48 dist-test-slave-hvgn krb5kdc[32013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:49:48.890684 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:41411
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:34491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:41411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:49.227200 32026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:49.227571 32026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:49.227726 32026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:49.234866 32026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:49.235011 32026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:49.235116 32026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:49.235230 32026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:49.235311 32026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:49.245067 32026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:34491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:41411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:41411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:49.246407 32026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:49.247709 32026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:49.259096 32034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:49.259130 32032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:49.259433 32031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:49.260317 32026 server_base.cc:1034] running on GCE node
I20260621 02:49:49.261101 32026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:49.262466 32026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:49.263731 32026 hybrid_clock.cc:648] HybridClock initialized: now 1782010189263687 us; error 68 us; skew 500 ppm
I20260621 02:49:49.264093 32026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:49 dist-test-slave-hvgn krb5kdc[32013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010189, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:49.268705 32026 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:49:49.270503 32026 webserver.cc:460] Webserver started at http://127.9.166.62:41247/ using document root <none> and password file <none>
I20260621 02:49:49.271420 32026 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:49.271559 32026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:49.271938 32026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:49.275645 32026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "5882059f5d9b4468bdbce3cf816946b2"
format_stamp: "Formatted at 2026-06-21 02:49:49 on dist-test-slave-hvgn"
server_key: "86c09b05d972c3ff375cba65e97c2b50"
server_key_iv: "c45c8879cdf9ad163698e4a89f58a6c9"
server_key_version: "encryptionkey@0"
I20260621 02:49:49.276412 32026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "5882059f5d9b4468bdbce3cf816946b2"
format_stamp: "Formatted at 2026-06-21 02:49:49 on dist-test-slave-hvgn"
server_key: "86c09b05d972c3ff375cba65e97c2b50"
server_key_iv: "c45c8879cdf9ad163698e4a89f58a6c9"
server_key_version: "encryptionkey@0"
I20260621 02:49:49.281469 32026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260621 02:49:49.285089 32041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:49.286470 32026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:49:49.286667 32026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "5882059f5d9b4468bdbce3cf816946b2"
format_stamp: "Formatted at 2026-06-21 02:49:49 on dist-test-slave-hvgn"
server_key: "86c09b05d972c3ff375cba65e97c2b50"
server_key_iv: "c45c8879cdf9ad163698e4a89f58a6c9"
server_key_version: "encryptionkey@0"
I20260621 02:49:49.286866 32026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:49.301123 32026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:49.305634 32026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:49.305919 32026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:49.324712 32026 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:41411
I20260621 02:49:49.324750 32092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:41411 every 8 connection(s)
I20260621 02:49:49.326186 32026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:49.329854 32093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:49.335465 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 32026
I20260621 02:49:49.335646 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:49.336093 9880 external_mini_cluster.cc:1428] Setting key aceab12ff358e9d51d76904fc356017a
I20260621 02:49:49.340687 32093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2: Bootstrap starting.
I20260621 02:49:49.344197 32093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:49.345424 32093 log.cc:824] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:49.348287 32093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2: No bootstrap required, opened a new log
I20260621 02:49:49.349574 32096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:49.338008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34701 (local address 127.9.166.62:41411)
0621 02:49:49.338862 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:49.338874 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:49.338914 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:49:49.340619 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:49.340644 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:49.341098 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:49.341424 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:49.342011 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:49.343317 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:49.345192 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:49.345902 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:49.346120 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:49.347304 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:49.347311 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:49.347325 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:49.347933 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:49.348001 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:49.348030 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:49.348172 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:49.348369 (+ 197us) server_negotiation.cc:299] Negotiation successful
0621 02:49:49.348625 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":420,"threads_started":1}
I20260621 02:49:49.353286 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:49:49.356305 32093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5882059f5d9b4468bdbce3cf816946b2" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41411 } }
I20260621 02:49:49.356686 32093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:49.356804 32093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5882059f5d9b4468bdbce3cf816946b2, State: Initialized, Role: FOLLOWER
I20260621 02:49:49.357467 32093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5882059f5d9b4468bdbce3cf816946b2" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41411 } }
I20260621 02:49:49.357756 32093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:49.357915 32093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:49.358174 32093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:49.361243 32093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5882059f5d9b4468bdbce3cf816946b2" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41411 } }
I20260621 02:49:49.361819 32093 leader_election.cc:302] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5882059f5d9b4468bdbce3cf816946b2; no voters:
I20260621 02:49:49.362564 32093 leader_election.cc:288] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:49.362677 32103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:49.365268 32096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:49.357865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45193 (local address 127.9.166.62:41411)
0621 02:49:49.358070 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:49.358079 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:49.358531 (+ 452us) server_negotiation.cc:407] Connection header received
0621 02:49:49.358747 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:49.358750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:49.358812 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:49.358904 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:49.359489 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:49.360235 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:49.362152 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:49.362810 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:49.363002 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:49.364426 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:49.364431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:49.364434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:49.364453 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:49.364493 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:49.364509 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:49.364657 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:49.364819 (+ 162us) server_negotiation.cc:299] Negotiation successful
0621 02:49:49.364916 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260621 02:49:49.365769 32093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:49.367702 32103 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2 [term 1 LEADER]: Becoming Leader. State: Replica: 5882059f5d9b4468bdbce3cf816946b2, State: Running, Role: LEADER
I20260621 02:49:49.368444 32103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5882059f5d9b4468bdbce3cf816946b2" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41411 } }
I20260621 02:49:49.371929 32105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5882059f5d9b4468bdbce3cf816946b2. Latest consensus state: current_term: 1 leader_uuid: "5882059f5d9b4468bdbce3cf816946b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5882059f5d9b4468bdbce3cf816946b2" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41411 } } }
I20260621 02:49:49.372009 32104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5882059f5d9b4468bdbce3cf816946b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5882059f5d9b4468bdbce3cf816946b2" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 41411 } } }
I20260621 02:49:49.372879 32104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:49.372364 32105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:49.375907 32110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:49.381204 32110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:49.390304 32110 catalog_manager.cc:1347] Generated new cluster ID: ac0cd287f5f84581958eeaf238aec8e0
I20260621 02:49:49.390434 32110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:49.417965 32110 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:49.419250 32110 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:49.427965 32110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5882059f5d9b4468bdbce3cf816946b2: Generated new TSK 0
I20260621 02:49:49.428833 32110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:49.445079 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 32026
W20260621 02:49:49.446190 32088 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34701
W20260621 02:49:49.446923 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 32026: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:49:49.447070 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 32026
2026-06-21T02:49:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1891 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:49 dist-test-slave-hvgn krb5kdc[32124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:49 dist-test-slave-hvgn krb5kdc[32124](info): set up 2 sockets
Jun 21 02:49:49 dist-test-slave-hvgn krb5kdc[32124](info): commencing operation
krb5kdc: starting...
W20260621 02:49:50.547923 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:50 dist-test-slave-hvgn krb5kdc[32124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:49:50.788765 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:37565
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:41955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:37565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:51.112457 32137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:51.112735 32137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:51.112892 32137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:51.120023 32137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:51.120126 32137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:51.120266 32137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:51.120364 32137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:51.120445 32137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:51.130260 32137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:41955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:37565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:37565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:51.131614 32137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:51.132884 32137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:51.144346 32145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:51.144435 32143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:51.144415 32142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:51.145116 32137 server_base.cc:1034] running on GCE node
I20260621 02:49:51.145879 32137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:51.147171 32137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:51.148366 32137 hybrid_clock.cc:648] HybridClock initialized: now 1782010191148313 us; error 49 us; skew 500 ppm
I20260621 02:49:51.148696 32137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:51 dist-test-slave-hvgn krb5kdc[32124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010191, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:51.156967 32137 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:49:51.158414 32137 webserver.cc:460] Webserver started at http://127.9.166.62:41765/ using document root <none> and password file <none>
I20260621 02:49:51.159015 32137 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:51.159111 32137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:51.159374 32137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:51.162279 32137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "e056c59000fa4b3789e56cafb2a1423b"
format_stamp: "Formatted at 2026-06-21 02:49:51 on dist-test-slave-hvgn"
server_key: "59129f16d7991aa6d2baa71d935b510c"
server_key_iv: "36dcaf9d1ef6d9423b4ca5d679ec90a7"
server_key_version: "encryptionkey@0"
I20260621 02:49:51.162966 32137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "e056c59000fa4b3789e56cafb2a1423b"
format_stamp: "Formatted at 2026-06-21 02:49:51 on dist-test-slave-hvgn"
server_key: "59129f16d7991aa6d2baa71d935b510c"
server_key_iv: "36dcaf9d1ef6d9423b4ca5d679ec90a7"
server_key_version: "encryptionkey@0"
I20260621 02:49:51.167958 32137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260621 02:49:51.171172 32152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:51.172502 32137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:49:51.172679 32137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "e056c59000fa4b3789e56cafb2a1423b"
format_stamp: "Formatted at 2026-06-21 02:49:51 on dist-test-slave-hvgn"
server_key: "59129f16d7991aa6d2baa71d935b510c"
server_key_iv: "36dcaf9d1ef6d9423b4ca5d679ec90a7"
server_key_version: "encryptionkey@0"
I20260621 02:49:51.172868 32137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:51.178536 32137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:51.182478 32137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:51.182762 32137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:51.201112 32137 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:37565
I20260621 02:49:51.201155 32203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:37565 every 8 connection(s)
I20260621 02:49:51.202492 32137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:51.206190 32204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:51.211310 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 32137
I20260621 02:49:51.211519 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:51.211952 9880 external_mini_cluster.cc:1428] Setting key 7338b53cfdb3308cf8908d37b9717b26
I20260621 02:49:51.217857 32204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b: Bootstrap starting.
I20260621 02:49:51.220700 32207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:51.213719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49375 (local address 127.9.166.62:37565)
0621 02:49:51.214568 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:51.214584 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:51.214638 (+ 54us) server_negotiation.cc:407] Connection header received
0621 02:49:51.216999 (+ 2361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:51.217025 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:51.217424 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:51.217881 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:51.218831 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:51.218839 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:51.218854 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:51.218870 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:51.218938 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:51.218963 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:51.219160 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:51.219285 (+ 125us) server_negotiation.cc:299] Negotiation successful
0621 02:49:51.219506 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":332,"threads_started":1}
I20260621 02:49:51.221586 32204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:51.222925 32204 log.cc:824] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:51.224422 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:49:51.226085 32204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b: No bootstrap required, opened a new log
I20260621 02:49:51.231961 32207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:51.229335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38853 (local address 127.9.166.62:37565)
0621 02:49:51.229586 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:51.229595 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:51.230139 (+ 544us) server_negotiation.cc:407] Connection header received
0621 02:49:51.230241 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:51.230248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:51.230314 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:51.230424 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:51.231295 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:51.231301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:51.231306 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0621 02:49:51.231324 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:51.231379 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0621 02:49:51.231421 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:51.231535 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:51.231601 (+ 66us) server_negotiation.cc:299] Negotiation successful
0621 02:49:51.231659 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:49:51.235476 32204 raft_consensus.cc:348] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e056c59000fa4b3789e56cafb2a1423b" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37565 } }
I20260621 02:49:51.235814 32204 raft_consensus.cc:374] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260621 02:49:51.235884 32204 raft_consensus.cc:729] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e056c59000fa4b3789e56cafb2a1423b, State: Initialized, Role: FOLLOWER
I20260621 02:49:51.236374 32204 consensus_queue.cc:260] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e056c59000fa4b3789e56cafb2a1423b" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37565 } }
I20260621 02:49:51.236580 32204 raft_consensus.cc:388] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260621 02:49:51.236672 32204 raft_consensus.cc:482] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260621 02:49:51.236800 32204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:51.238994 32204 raft_consensus.cc:504] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e056c59000fa4b3789e56cafb2a1423b" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37565 } }
I20260621 02:49:51.239614 32204 leader_election.cc:302] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e056c59000fa4b3789e56cafb2a1423b; no voters:
I20260621 02:49:51.240185 32204 leader_election.cc:288] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260621 02:49:51.240340 32214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:51.240989 32214 raft_consensus.cc:686] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b [term 1 LEADER]: Becoming Leader. State: Replica: e056c59000fa4b3789e56cafb2a1423b, State: Running, Role: LEADER
I20260621 02:49:51.241528 32214 consensus_queue.cc:237] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e056c59000fa4b3789e56cafb2a1423b" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37565 } }
I20260621 02:49:51.242223 32204 sys_catalog.cc:564] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:51.244406 32215 sys_catalog.cc:455] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e056c59000fa4b3789e56cafb2a1423b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e056c59000fa4b3789e56cafb2a1423b" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37565 } } }
I20260621 02:49:51.244856 32215 sys_catalog.cc:458] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:51.248503 32216 sys_catalog.cc:455] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e056c59000fa4b3789e56cafb2a1423b. Latest consensus state: current_term: 1 leader_uuid: "e056c59000fa4b3789e56cafb2a1423b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e056c59000fa4b3789e56cafb2a1423b" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37565 } } }
I20260621 02:49:51.248852 32216 sys_catalog.cc:458] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:51.249629 32221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:51.254040 32221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:51.262630 32221 catalog_manager.cc:1347] Generated new cluster ID: de3289c160f14ddda5fe47b38d09e831
I20260621 02:49:51.262742 32221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:51.301323 32221 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:51.301790 32221 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:51.311535 32221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e056c59000fa4b3789e56cafb2a1423b: Generated new TSK 0
I20260621 02:49:51.312340 32221 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:51.324333 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 32137
W20260621 02:49:51.325585 32199 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49375
W20260621 02:49:51.326287 9880 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 32137: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260621 02:49:51.326423 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 32137
2026-06-21T02:49:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1879 ms)
[----------] 18 tests from AuthTokenIssuingTest (34473 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:51 dist-test-slave-hvgn krb5kdc[32235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:51 dist-test-slave-hvgn krb5kdc[32235](info): set up 2 sockets
Jun 21 02:49:51 dist-test-slave-hvgn krb5kdc[32235](info): commencing operation
krb5kdc: starting...
W20260621 02:49:52.431917 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:52 dist-test-slave-hvgn krb5kdc[32235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:49:52.682427 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:37277
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:44581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:37277,127.9.166.61:41743,127.9.166.60:40153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:53.007869 32248 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 02:49:53.008190 32248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:53.008350 32248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:53.008486 32248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:53.015735 32248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:53.015816 32248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:53.015919 32248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:53.016006 32248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:53.016119 32248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:53.025765 32248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:44581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:37277,127.9.166.61:41743,127.9.166.60:40153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:37277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:53.027015 32248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:53.028306 32248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:53.039892 32253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:53.039929 32256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:53.039853 32254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:53.040962 32248 server_base.cc:1034] running on GCE node
I20260621 02:49:53.041749 32248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:53.043057 32248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:53.044272 32248 hybrid_clock.cc:648] HybridClock initialized: now 1782010193044219 us; error 30 us; skew 500 ppm
I20260621 02:49:53.044556 32248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:53 dist-test-slave-hvgn krb5kdc[32235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010193, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:53.047710 32248 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:49:53.049077 32248 webserver.cc:460] Webserver started at http://127.9.166.62:37245/ using document root <none> and password file <none>
I20260621 02:49:53.049767 32248 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:53.049883 32248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:53.050213 32248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:53.053124 32248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "b0129aa3f60f411e9d9378703d8c04d6"
format_stamp: "Formatted at 2026-06-21 02:49:53 on dist-test-slave-hvgn"
server_key: "497bb01804539e9808c3dea3ae41052d"
server_key_iv: "6ad1c477cc389b0b040e9f0049cff094"
server_key_version: "encryptionkey@0"
I20260621 02:49:53.053890 32248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "b0129aa3f60f411e9d9378703d8c04d6"
format_stamp: "Formatted at 2026-06-21 02:49:53 on dist-test-slave-hvgn"
server_key: "497bb01804539e9808c3dea3ae41052d"
server_key_iv: "6ad1c477cc389b0b040e9f0049cff094"
server_key_version: "encryptionkey@0"
I20260621 02:49:53.058795 32248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260621 02:49:53.062127 32263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:53.063500 32248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:49:53.063681 32248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "b0129aa3f60f411e9d9378703d8c04d6"
format_stamp: "Formatted at 2026-06-21 02:49:53 on dist-test-slave-hvgn"
server_key: "497bb01804539e9808c3dea3ae41052d"
server_key_iv: "6ad1c477cc389b0b040e9f0049cff094"
server_key_version: "encryptionkey@0"
I20260621 02:49:53.063885 32248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:53.079495 32248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:53.083158 32248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:53.083465 32248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:53.102056 32248 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:37277
I20260621 02:49:53.102064 32314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:37277 every 8 connection(s)
I20260621 02:49:53.103498 32248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:53.106340 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 32248
I20260621 02:49:53.106577 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:53.107273 9880 external_mini_cluster.cc:1428] Setting key 63519a322e79b4b222e9f489846b2f07
I20260621 02:49:53.107278 32315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:53.113574 32315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:49:53 dist-test-slave-hvgn krb5kdc[32235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010193, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.166.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.61@KRBTEST.COM" created.
I20260621 02:49:53.138789 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.116650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37277 (local address 127.0.0.1:51720)
0621 02:49:53.117628 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.117686 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:53.120405 (+ 2719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:53.121376 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:53.121381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:53.121917 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:53.122255 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:53.122272 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:53.124877 (+ 2605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:53.124881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:53.125944 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:53.125955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:53.127492 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:53.127498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:53.127684 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:53.128367 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:53.128388 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:53.130631 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:53.134705 (+ 4074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:53.134714 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:53.134723 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:53.135189 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:53.135758 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:53.135765 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:53.135772 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:53.135948 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:53.136950 (+ 1002us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:53.136956 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:53.137153 (+ 197us) client_negotiation.cc:769] Sending connection context
0621 02:49:53.137368 (+ 215us) client_negotiation.cc:240] Negotiation successful
0621 02:49:53.137743 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":47,"thread_start_us":396,"threads_started":1}
I20260621 02:49:53.138815 32319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.116650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51720 (local address 127.9.166.62:37277)
0621 02:49:53.120021 (+ 3371us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:53.120032 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:53.120088 (+ 56us) server_negotiation.cc:407] Connection header received
0621 02:49:53.120786 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:53.120799 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:53.121108 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:53.121334 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:53.122473 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:53.124715 (+ 2242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:53.126097 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:53.127221 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:53.127478 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:53.130882 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:53.130893 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:53.130910 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:53.130956 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:53.134592 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:53.135427 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:53.135436 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:53.135441 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:53.135643 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:53.136060 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:53.136065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:53.136068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:53.136809 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:53.136998 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:53.137547 (+ 549us) server_negotiation.cc:299] Negotiation successful
0621 02:49:53.137918 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":3118,"spinlock_wait_cycles":34176,"thread_start_us":2846,"threads_started":1}
I20260621 02:49:53.141846 32315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:49:53.142912 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.142536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:41743 (local address 127.0.0.1:51870)
0621 02:49:53.142699 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.142809 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260621 02:49:53.143256 32267 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.166.61:41743: connect: Connection refused (error 111)
I20260621 02:49:53.144779 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.144435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:41743 (local address 127.0.0.1:51884)
0621 02:49:53.144582 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.144641 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260621 02:49:53.144982 32315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.61:41743: Network error: Client connection negotiation failed: client connection to 127.9.166.61:41743: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.61.keytab.
Entry for principal kudu/127.9.166.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.61.keytab.
I20260621 02:49:53.164741 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.61:41743
--webserver_interface=127.9.166.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.61.keytab
--principal=kudu/127.9.166.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:44581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:37277,127.9.166.61:41743,127.9.166.60:40153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:49:53.194345 32315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } attempt: 1
I20260621 02:49:53.195572 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.195014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:41743 (local address 127.0.0.1:51896)
0621 02:49:53.195227 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.195346 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260621 02:49:53.196600 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.196224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:41743 (local address 127.0.0.1:51908)
0621 02:49:53.196366 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.196440 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260621 02:49:53.196826 32315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.61:41743: Network error: Client connection negotiation failed: client connection to 127.9.166.61:41743: connect: Connection refused (error 111)
I20260621 02:49:53.265092 32315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } attempt: 2
I20260621 02:49:53.266592 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.265874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:41743 (local address 127.0.0.1:51914)
0621 02:49:53.266192 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.266329 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260621 02:49:53.267943 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.267442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:41743 (local address 127.0.0.1:51922)
0621 02:49:53.267615 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.267762 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260621 02:49:53.268247 32315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.61:41743: Network error: Client connection negotiation failed: client connection to 127.9.166.61:41743: connect: Connection refused (error 111)
I20260621 02:49:53.359531 32315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } attempt: 3
I20260621 02:49:53.360733 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.360276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:41743 (local address 127.0.0.1:51926)
0621 02:49:53.360488 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.360582 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260621 02:49:53.361757 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.361383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:41743 (local address 127.0.0.1:51942)
0621 02:49:53.361531 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.361615 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260621 02:49:53.361966 32315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.61:41743: Network error: Client connection negotiation failed: client connection to 127.9.166.61:41743: connect: Connection refused (error 111)
W20260621 02:49:53.486579 32322 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 02:49:53.486904 32322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:53.487032 32322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:53.487174 32322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:53.494338 32322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:53.494424 32322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:53.494524 32322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:53.494598 32322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:53.494683 32322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:53.504423 32322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:44581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:37277,127.9.166.61:41743,127.9.166.60:40153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.61.keytab
--principal=kudu/127.9.166.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.61:41743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.166.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:53.505240 32315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } attempt: 4
I20260621 02:49:53.505812 32322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:53.506438 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.505899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:41743 (local address 127.0.0.1:51948)
0621 02:49:53.506217 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.506315 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260621 02:49:53.507294 32322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 02:49:53.507465 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.507074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:41743 (local address 127.0.0.1:51952)
0621 02:49:53.507213 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.507298 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 02:49:53.507640 32315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.61:41743: Network error: Client connection negotiation failed: client connection to 127.9.166.61:41743: connect: Connection refused (error 111)
W20260621 02:49:53.519742 32327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:53.519785 32330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:53.519841 32328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:53.520617 32322 server_base.cc:1034] running on GCE node
I20260621 02:49:53.521414 32322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:53.522791 32322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:53.524061 32322 hybrid_clock.cc:648] HybridClock initialized: now 1782010193523984 us; error 51 us; skew 500 ppm
I20260621 02:49:53.524417 32322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:53 dist-test-slave-hvgn krb5kdc[32235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010193, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:53.527626 32322 init.cc:377] Logged in from keytab as kudu/127.9.166.61@KRBTEST.COM (short username kudu)
I20260621 02:49:53.529034 32322 webserver.cc:460] Webserver started at http://127.9.166.61:36399/ using document root <none> and password file <none>
I20260621 02:49:53.529649 32322 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:53.529731 32322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:53.530048 32322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:53.532917 32322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-1/data/instance:
uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6"
format_stamp: "Formatted at 2026-06-21 02:49:53 on dist-test-slave-hvgn"
server_key: "b3a7385d8bb85d30b823b21f479751e1"
server_key_iv: "f3f39e46081da3b900800b1caca9bd86"
server_key_version: "encryptionkey@0"
I20260621 02:49:53.533643 32322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-1/wal/instance:
uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6"
format_stamp: "Formatted at 2026-06-21 02:49:53 on dist-test-slave-hvgn"
server_key: "b3a7385d8bb85d30b823b21f479751e1"
server_key_iv: "f3f39e46081da3b900800b1caca9bd86"
server_key_version: "encryptionkey@0"
I20260621 02:49:53.538653 32322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260621 02:49:53.542086 32337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:53.543548 32322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:49:53.543718 32322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-1/wal
uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6"
format_stamp: "Formatted at 2026-06-21 02:49:53 on dist-test-slave-hvgn"
server_key: "b3a7385d8bb85d30b823b21f479751e1"
server_key_iv: "f3f39e46081da3b900800b1caca9bd86"
server_key_version: "encryptionkey@0"
I20260621 02:49:53.543941 32322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:53.560899 32322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:53.565414 32322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:53.565681 32322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:53.585099 32322 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.61:41743
I20260621 02:49:53.585119 32388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.61:41743 every 8 connection(s)
I20260621 02:49:53.586452 32322 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-1/data/info.pb
I20260621 02:49:53.587864 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 32322
I20260621 02:49:53.588054 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-1/wal/instance
I20260621 02:49:53.588598 9880 external_mini_cluster.cc:1428] Setting key 998d1277a192771a920998356dbd7bcb
I20260621 02:49:53.591177 32389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:53.600615 32389 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.166.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.60@KRBTEST.COM" created.
Jun 21 02:49:53 dist-test-slave-hvgn krb5kdc[32235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010193, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.61@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:49:53.624675 32319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.603187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51734 (local address 127.9.166.62:37277)
0621 02:49:53.603441 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:53.603450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:53.604626 (+ 1176us) server_negotiation.cc:407] Connection header received
0621 02:49:53.606532 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:53.606541 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:53.606638 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:53.606770 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:53.608349 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:53.609545 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:53.611182 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:53.612023 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:53.612164 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:53.617592 (+ 5428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:53.617601 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:53.617609 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:53.617658 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:53.621332 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:53.622051 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:53.622059 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:53.622065 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:53.622212 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:53.622631 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:53.622638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:53.622644 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:53.623053 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:53.623217 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:53.623982 (+ 765us) server_negotiation.cc:299] Negotiation successful
0621 02:49:53.624236 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:49:53.625428 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.603657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37277 (local address 127.0.0.1:51734)
0621 02:49:53.604447 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.604500 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:53.606266 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:53.606953 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:53.606961 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:53.607604 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:53.608158 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:53.608175 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:53.609714 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:53.609718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:53.611015 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:53.611031 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:53.612272 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:53.612276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:53.612464 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:53.613189 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:53.613211 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:53.617347 (+ 4136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:53.621477 (+ 4130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:53.621484 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:53.621498 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:53.621915 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:53.622332 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:53.622335 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:53.622338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:53.622511 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:53.623202 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:53.623208 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:53.623598 (+ 390us) client_negotiation.cc:769] Sending connection context
0621 02:49:53.623830 (+ 232us) client_negotiation.cc:240] Negotiation successful
0621 02:49:53.624230 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":335,"threads_started":1}
I20260621 02:49:53.627183 32389 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:49:53 dist-test-slave-hvgn krb5kdc[32235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010193, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.61@KRBTEST.COM for kudu/127.9.166.61@KRBTEST.COM
I20260621 02:49:53.642634 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.628016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:41743 (local address 127.0.0.1:51958)
0621 02:49:53.628249 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.628269 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:53.628379 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:53.629157 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:53.629164 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:53.629639 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:53.629846 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:53.629859 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:53.631277 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:53.631285 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:53.632898 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:53.632915 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:53.633712 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:53.633721 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:53.633933 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:53.634540 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:53.634568 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:53.637040 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:53.640223 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:53.640228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:53.640231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:53.640577 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:53.640954 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:53.640958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:53.640962 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:53.641083 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:53.641648 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:53.641652 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:53.641805 (+ 153us) client_negotiation.cc:769] Sending connection context
0621 02:49:53.641955 (+ 150us) client_negotiation.cc:240] Negotiation successful
0621 02:49:53.642223 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260621 02:49:53.642726 32394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.628111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51958 (local address 127.9.166.61:41743)
0621 02:49:53.628726 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:53.628732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:53.628772 (+ 40us) server_negotiation.cc:407] Connection header received
0621 02:49:53.628816 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:53.628822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:53.628989 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:53.629102 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:53.630061 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:53.631138 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:53.633040 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:53.633556 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:53.633650 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:53.637245 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:53.637259 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:53.637274 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:53.637302 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:53.640093 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:53.640703 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:53.640718 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:53.640720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:53.640831 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:53.641183 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:53.641190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:53.641193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:53.641525 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:53.641753 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:53.642127 (+ 374us) server_negotiation.cc:299] Negotiation successful
0621 02:49:53.642341 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":333,"threads_started":1}
I20260621 02:49:53.645435 32389 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:49:53.646908 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.646305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46290)
0621 02:49:53.646545 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.646699 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260621 02:49:53.647178 32340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
I20260621 02:49:53.649010 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.648657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46292)
0621 02:49:53.648801 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.648889 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.60.keytab.
Entry for principal kudu/127.9.166.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.60.keytab.
W20260621 02:49:53.649233 32389 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:40153: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
I20260621 02:49:53.650354 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.60:40153
--webserver_interface=127.9.166.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.60.keytab
--principal=kudu/127.9.166.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:44581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:37277,127.9.166.61:41743,127.9.166.60:40153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:49:53.698566 32389 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } attempt: 1
I20260621 02:49:53.699908 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.699366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46304)
0621 02:49:53.699622 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.699751 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260621 02:49:53.700953 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.700605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46308)
0621 02:49:53.700768 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.700852 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260621 02:49:53.701256 32389 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:40153: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
I20260621 02:49:53.769457 32389 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } attempt: 2
I20260621 02:49:53.770560 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.770094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46312)
0621 02:49:53.770300 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.770397 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260621 02:49:53.771589 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.771251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46314)
0621 02:49:53.771413 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.771484 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260621 02:49:53.771833 32389 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:40153: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
I20260621 02:49:53.806905 32315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } attempt: 5
Jun 21 02:49:53 dist-test-slave-hvgn krb5kdc[32235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010193, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for kudu/127.9.166.61@KRBTEST.COM
I20260621 02:49:53.821748 32394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.807854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51972 (local address 127.9.166.61:41743)
0621 02:49:53.808044 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:53.808052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:53.808152 (+ 100us) server_negotiation.cc:407] Connection header received
0621 02:49:53.808394 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:53.808400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:53.808491 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:53.808596 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:53.809403 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:53.810195 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:53.811345 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:53.811874 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:53.812029 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:53.815768 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:53.815775 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:53.815780 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:53.815822 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:53.819119 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:53.819861 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:53.819866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:53.819870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:53.820002 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:53.820386 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:53.820391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:53.820395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:53.820669 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:53.820825 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:53.821197 (+ 372us) server_negotiation.cc:299] Negotiation successful
0621 02:49:53.821415 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260621 02:49:53.821762 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.807807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:41743 (local address 127.0.0.1:51972)
0621 02:49:53.808031 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.808053 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:53.808233 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:53.808631 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:53.808636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:53.809057 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:53.809234 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:53.809243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:53.810324 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:53.810332 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:53.811186 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:53.811197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:53.812047 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:53.812054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:53.812284 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:53.812938 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:53.812956 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:53.815500 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:53.819264 (+ 3764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:53.819269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:53.819273 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:53.819733 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:53.820130 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:53.820135 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:53.820139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:53.820280 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:53.820792 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:53.820797 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:53.820956 (+ 159us) client_negotiation.cc:769] Sending connection context
0621 02:49:53.821101 (+ 145us) client_negotiation.cc:240] Negotiation successful
0621 02:49:53.821393 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260621 02:49:53.823191 32315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:49:53.824294 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.823919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46328)
0621 02:49:53.824080 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.824158 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260621 02:49:53.825197 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.824884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46336)
0621 02:49:53.825021 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.825094 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260621 02:49:53.825423 32315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:40153: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
I20260621 02:49:53.863054 32389 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } attempt: 3
I20260621 02:49:53.864260 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.863750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46342)
0621 02:49:53.863967 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.864075 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260621 02:49:53.865389 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.864964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46348)
0621 02:49:53.865188 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.865294 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260621 02:49:53.865651 32389 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:40153: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
I20260621 02:49:53.876683 32315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } attempt: 1
I20260621 02:49:53.877723 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.877271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46352)
0621 02:49:53.877478 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.877566 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260621 02:49:53.878698 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.878373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46354)
0621 02:49:53.878526 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.878595 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260621 02:49:53.878908 32315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:40153: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
I20260621 02:49:53.947150 32315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } attempt: 2
I20260621 02:49:53.948335 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.947865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46356)
0621 02:49:53.948060 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.948176 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260621 02:49:53.949327 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:53.949016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46364)
0621 02:49:53.949152 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:53.949222 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 02:49:53.949548 32315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:40153: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
W20260621 02:49:53.977424 32396 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 02:49:53.977814 32396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:53.977944 32396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:53.978085 32396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:53.985488 32396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:53.985605 32396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:53.985707 32396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:53.985800 32396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:53.985888 32396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:53.995749 32396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:44581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:37277,127.9.166.61:41743,127.9.166.60:40153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.60.keytab
--principal=kudu/127.9.166.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.60:40153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.166.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:53.997058 32396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:53.998412 32396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 02:49:54.008853 32389 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } attempt: 4
I20260621 02:49:54.010115 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:54.009543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46366)
0621 02:49:54.009779 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:54.009896 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260621 02:49:54.010236 32401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:54.010926 32402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:54.011161 32404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:54.011279 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:54.010850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46368)
0621 02:49:54.011062 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:54.011149 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260621 02:49:54.011544 32389 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:40153: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
I20260621 02:49:54.012590 32396 server_base.cc:1034] running on GCE node
I20260621 02:49:54.013437 32396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:54.014886 32396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:54.016122 32396 hybrid_clock.cc:648] HybridClock initialized: now 1782010194016056 us; error 43 us; skew 500 ppm
I20260621 02:49:54.016479 32396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:54 dist-test-slave-hvgn krb5kdc[32235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010194, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:54.019620 32396 init.cc:377] Logged in from keytab as kudu/127.9.166.60@KRBTEST.COM (short username kudu)
I20260621 02:49:54.021129 32396 webserver.cc:460] Webserver started at http://127.9.166.60:36443/ using document root <none> and password file <none>
I20260621 02:49:54.021807 32396 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:54.021938 32396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:54.022264 32396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:54.025281 32396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-2/data/instance:
uuid: "b1e50122c73046138f5d23aa6860b6e1"
format_stamp: "Formatted at 2026-06-21 02:49:54 on dist-test-slave-hvgn"
server_key: "eed9437c1c8d560411a629391f1352ab"
server_key_iv: "a869181bc28e5b4114d3046bdb4da5dd"
server_key_version: "encryptionkey@0"
I20260621 02:49:54.026201 32396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-2/wal/instance:
uuid: "b1e50122c73046138f5d23aa6860b6e1"
format_stamp: "Formatted at 2026-06-21 02:49:54 on dist-test-slave-hvgn"
server_key: "eed9437c1c8d560411a629391f1352ab"
server_key_iv: "a869181bc28e5b4114d3046bdb4da5dd"
server_key_version: "encryptionkey@0"
I20260621 02:49:54.031363 32396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260621 02:49:54.034719 32411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:54.036046 32396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 02:49:54.036219 32396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-2/wal
uuid: "b1e50122c73046138f5d23aa6860b6e1"
format_stamp: "Formatted at 2026-06-21 02:49:54 on dist-test-slave-hvgn"
server_key: "eed9437c1c8d560411a629391f1352ab"
server_key_iv: "a869181bc28e5b4114d3046bdb4da5dd"
server_key_version: "encryptionkey@0"
I20260621 02:49:54.036438 32396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:54.054795 32396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:54.055758 32315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } attempt: 3
I20260621 02:49:54.056823 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:54.056326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46382)
0621 02:49:54.056549 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:54.056636 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260621 02:49:54.057901 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:54.057517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46390)
0621 02:49:54.057711 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:54.057781 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260621 02:49:54.058123 32315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:40153: Network error: Client connection negotiation failed: client connection to 127.9.166.60:40153: connect: Connection refused (error 111)
I20260621 02:49:54.059363 32396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:54.059675 32396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:54.078006 32396 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.60:40153
I20260621 02:49:54.078029 32462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.60:40153 every 8 connection(s)
I20260621 02:49:54.079311 32396 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-2/data/info.pb
I20260621 02:49:54.082858 32463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:54.084981 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 32396
I20260621 02:49:54.085165 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1782009844354615-9880-0/minicluster-data/master-2/wal/instance
I20260621 02:49:54.085651 9880 external_mini_cluster.cc:1428] Setting key c4f3695636a77c2e3b8c031335397881
I20260621 02:49:54.090380 32463 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:49:54 dist-test-slave-hvgn krb5kdc[32235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.60@KRBTEST.COM
Jun 21 02:49:54 dist-test-slave-hvgn krb5kdc[32235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010194, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.60@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:49:54.108608 32319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:54.092796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51742 (local address 127.9.166.62:37277)
0621 02:49:54.093011 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.093017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.093607 (+ 590us) server_negotiation.cc:407] Connection header received
0621 02:49:54.093807 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.093813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.093895 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.093973 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.095301 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.096354 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.098367 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.098767 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.098857 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:54.102493 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:54.102499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:54.102502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:54.102533 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:54.105480 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.106286 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.106293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.106298 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.106473 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.106877 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.106889 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.106895 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.107152 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:54.107336 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:54.107991 (+ 655us) server_negotiation.cc:299] Negotiation successful
0621 02:49:54.108213 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:49:54.109664 32467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:54.092717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:37277 (local address 127.0.0.1:51742)
0621 02:49:54.093463 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:54.093509 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:54.093662 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:54.094063 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:54.094070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:54.094852 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:54.095162 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:54.095173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.096517 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.096521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:54.098203 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:54.098221 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.099002 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.099011 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:54.099197 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:54.099797 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:54.099817 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:54.102255 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:54.105624 (+ 3369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:54.105634 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:54.105646 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:54.106177 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:54.106589 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:54.106594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:54.106597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:54.106765 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:54.107296 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:54.107306 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:54.107588 (+ 282us) client_negotiation.cc:769] Sending connection context
0621 02:49:54.108806 (+ 1218us) client_negotiation.cc:240] Negotiation successful
0621 02:49:54.109079 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":365,"threads_started":1}
I20260621 02:49:54.109726 32466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:54.087519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46406 (local address 127.9.166.60:40153)
0621 02:49:54.088233 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.088244 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.088283 (+ 39us) server_negotiation.cc:407] Connection header received
0621 02:49:54.089911 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.089937 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.090294 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.090691 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.092143 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.093671 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.095536 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.096494 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.096631 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:54.101313 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:54.101322 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:54.101336 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:54.101381 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:54.105115 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.105933 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.105940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.105942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.106097 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.106567 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.106572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.106575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.107307 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:54.107658 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:54.107937 (+ 279us) server_negotiation.cc:299] Negotiation successful
0621 02:49:54.108357 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"spinlock_wait_cycles":21888,"thread_start_us":305,"threads_started":1}
I20260621 02:49:54.111693 32463 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:49:54 dist-test-slave-hvgn krb5kdc[32235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010194, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.60@KRBTEST.COM for kudu/127.9.166.61@KRBTEST.COM
Jun 21 02:49:54 dist-test-slave-hvgn krb5kdc[32235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:49:54.125170 32467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:54.112500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:41743 (local address 127.0.0.1:51976)
0621 02:49:54.112679 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:54.112698 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:54.112793 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:54.113545 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:54.113551 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:54.113987 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:54.114160 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:54.114171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.115107 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.115112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:54.116093 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:54.116105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.116851 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.116856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:54.116983 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:54.117524 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:54.117541 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:54.119565 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:54.122658 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:54.122664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:54.122668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:54.123199 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:54.123622 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:54.123629 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:54.123634 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:54.123831 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:54.124309 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:54.124314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:54.124490 (+ 176us) client_negotiation.cc:769] Sending connection context
0621 02:49:54.124677 (+ 187us) client_negotiation.cc:240] Negotiation successful
0621 02:49:54.124935 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260621 02:49:54.125409 32394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:54.113006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51976 (local address 127.9.166.61:41743)
0621 02:49:54.113232 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.113239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.113257 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:49:54.113321 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.113326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.113406 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.113505 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.114309 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.114963 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.116245 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.116716 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.116830 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:54.119750 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:54.119755 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:54.119759 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:54.119790 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:54.122539 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.123312 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.123333 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.123335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.123515 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.123938 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.123941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.123943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.124184 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:54.124298 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:54.124791 (+ 493us) server_negotiation.cc:299] Negotiation successful
0621 02:49:54.125012 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260621 02:49:54.126524 32463 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:49:54.127485 32319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:54.112985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51754 (local address 127.9.166.62:37277)
0621 02:49:54.113152 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.113162 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.113306 (+ 144us) server_negotiation.cc:407] Connection header received
0621 02:49:54.113435 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.113438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.113489 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.113592 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.114683 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.115452 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.116622 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.117175 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.117263 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:54.121477 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:54.121485 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:54.121490 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:54.121534 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:54.124652 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.125440 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.125446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.125450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.125613 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.126026 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.126031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.126034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.126345 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:54.126625 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:54.126906 (+ 281us) server_negotiation.cc:299] Negotiation successful
0621 02:49:54.127121 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Jun 21 02:49:54 dist-test-slave-hvgn krb5kdc[32235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010194, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.60@KRBTEST.COM for kudu/127.9.166.60@KRBTEST.COM
Jun 21 02:49:54 dist-test-slave-hvgn krb5kdc[32235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.61@KRBTEST.COM
I20260621 02:49:54.140087 32467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:54.127327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46408)
0621 02:49:54.127551 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:54.127571 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:54.127677 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:54.127996 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:54.128000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:54.128361 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:54.128478 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:54.128485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.129543 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.129548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:54.130466 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:54.130477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.131198 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.131204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:54.131447 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:54.132013 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:54.132034 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:54.133980 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:54.137461 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:54.137466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:54.137469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:54.137962 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:54.138328 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:54.138335 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:54.138338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:54.138468 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:54.138964 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:54.138968 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:54.139169 (+ 201us) client_negotiation.cc:769] Sending connection context
0621 02:49:54.139348 (+ 179us) client_negotiation.cc:240] Negotiation successful
0621 02:49:54.139661 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260621 02:49:54.140281 32466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:54.127584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.9.166.60:40153)
0621 02:49:54.127737 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.127743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.127758 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:49:54.127800 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.127805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.127867 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.127982 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.128614 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.129406 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.130596 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.131072 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.131210 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:54.134143 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:54.134153 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:54.134160 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:54.134208 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:54.137336 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.138068 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.138071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.138073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.138184 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.138565 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.138573 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.138576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.138816 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:54.138983 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:54.139487 (+ 504us) server_negotiation.cc:299] Negotiation successful
0621 02:49:54.139840 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":59}
I20260621 02:49:54.144037 32394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:54.129024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51980 (local address 127.9.166.61:41743)
0621 02:49:54.129193 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.129199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.129216 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:49:54.129328 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.129333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.129397 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.129492 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.130617 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.131480 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.132609 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.133156 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.133293 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:54.138395 (+ 5102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:54.138399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:54.138402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:54.138430 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:54.141612 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.142380 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.142384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.142386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.142509 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.142875 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.142878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.142880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.143099 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:54.143291 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:54.143623 (+ 332us) server_negotiation.cc:299] Negotiation successful
0621 02:49:54.143776 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260621 02:49:54.146317 32463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1: Bootstrap starting.
I20260621 02:49:54.149035 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:49:54.149906 32463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:54.150909 32463 log.cc:824] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:54.153949 32463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1: No bootstrap required, opened a new log
I20260621 02:49:54.151734 32290 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b0129aa3f60f411e9d9378703d8c04d6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51754
I20260621 02:49:54.164579 32463 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0129aa3f60f411e9d9378703d8c04d6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } } peers { permanent_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } } peers { permanent_uuid: "b1e50122c73046138f5d23aa6860b6e1" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } }
I20260621 02:49:54.164978 32463 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e50122c73046138f5d23aa6860b6e1, State: Initialized, Role: FOLLOWER
I20260621 02:49:54.165470 32463 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0129aa3f60f411e9d9378703d8c04d6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } } peers { permanent_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } } peers { permanent_uuid: "b1e50122c73046138f5d23aa6860b6e1" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } }
I20260621 02:49:54.166343 32469 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0129aa3f60f411e9d9378703d8c04d6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } } peers { permanent_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } } peers { permanent_uuid: "b1e50122c73046138f5d23aa6860b6e1" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } } }
I20260621 02:49:54.166666 32469 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:49:54.167567 32463 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1 [sys.catalog]: configured and running, proceeding with master startup.
W20260621 02:49:54.176841 32480 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:49:54.176959 32480 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:49:54.174801 32396 master_runner.cc:186] Error getting master registration for 127.9.166.62:37277: OK, instance_id { permanent_uuid: "b0129aa3f60f411e9d9378703d8c04d6" instance_seqno: 1782010193096644 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 02:49:54.185732 32396 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 02:49:54.235369 32315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } attempt: 4
Jun 21 02:49:54 dist-test-slave-hvgn krb5kdc[32235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010193, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for kudu/127.9.166.60@KRBTEST.COM
I20260621 02:49:54.251987 32466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:54.236306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46412 (local address 127.9.166.60:40153)
0621 02:49:54.236527 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.236533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.236556 (+ 23us) server_negotiation.cc:407] Connection header received
0621 02:49:54.236719 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.236724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.236806 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.236914 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.237748 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.238928 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.240378 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.241185 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.241309 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:54.245579 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:54.245585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:54.245590 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:54.245631 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:54.249248 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.249996 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.250000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.250002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.250131 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.250606 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.250611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.250617 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.250974 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:54.251106 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:54.251513 (+ 407us) server_negotiation.cc:299] Negotiation successful
0621 02:49:54.251699 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260621 02:49:54.252115 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:54.236205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46412)
0621 02:49:54.236385 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:54.236408 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:54.236553 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:54.236941 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:54.236945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:54.237410 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:54.237581 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:54.237591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.239077 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.239083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:54.240236 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:54.240249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.241344 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.241350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:54.241560 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:54.242283 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:54.242308 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:54.245366 (+ 3058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:54.249363 (+ 3997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:54.249372 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:54.249376 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:54.249869 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:54.250278 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:54.250283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:54.250287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:54.250455 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:54.251063 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:54.251068 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:54.251230 (+ 162us) client_negotiation.cc:769] Sending connection context
0621 02:49:54.251404 (+ 174us) client_negotiation.cc:240] Negotiation successful
0621 02:49:54.251708 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260621 02:49:54.257011 32315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6: Bootstrap starting.
I20260621 02:49:54.260308 32315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:54.261387 32315 log.cc:824] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:54.264256 32315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6: No bootstrap required, opened a new log
I20260621 02:49:54.270145 32315 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0129aa3f60f411e9d9378703d8c04d6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } } peers { permanent_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } } peers { permanent_uuid: "b1e50122c73046138f5d23aa6860b6e1" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } }
I20260621 02:49:54.270545 32315 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0129aa3f60f411e9d9378703d8c04d6, State: Initialized, Role: FOLLOWER
I20260621 02:49:54.271015 32315 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0129aa3f60f411e9d9378703d8c04d6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } } peers { permanent_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } } peers { permanent_uuid: "b1e50122c73046138f5d23aa6860b6e1" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } }
I20260621 02:49:54.272042 32483 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0129aa3f60f411e9d9378703d8c04d6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } } peers { permanent_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } } peers { permanent_uuid: "b1e50122c73046138f5d23aa6860b6e1" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } } }
I20260621 02:49:54.272393 32483 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:49:54.273164 32315 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:54.273257 32290 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260621 02:49:54.273459 32290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:54.276746 32290 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0129aa3f60f411e9d9378703d8c04d6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } } peers { permanent_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } } peers { permanent_uuid: "b1e50122c73046138f5d23aa6860b6e1" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } }
I20260621 02:49:54.278254 32290 leader_election.cc:288] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [CANDIDATE]: Term 1 election: Requested vote from peers 60b8aa64436a4033a7cea3dbcfbb7bf6 (127.9.166.61:41743), b1e50122c73046138f5d23aa6860b6e1 (127.9.166.60:40153)
I20260621 02:49:54.279371 32438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b0129aa3f60f411e9d9378703d8c04d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b1e50122c73046138f5d23aa6860b6e1"
I20260621 02:49:54.279973 32438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:54.284526 32438 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0129aa3f60f411e9d9378703d8c04d6 in term 1.
I20260621 02:49:54.285311 32266 leader_election.cc:302] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0129aa3f60f411e9d9378703d8c04d6, b1e50122c73046138f5d23aa6860b6e1; no voters:
I20260621 02:49:54.285910 32483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:54.296981 32483 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [term 1 LEADER]: Becoming Leader. State: Replica: b0129aa3f60f411e9d9378703d8c04d6, State: Running, Role: LEADER
I20260621 02:49:54.297770 32483 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0129aa3f60f411e9d9378703d8c04d6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } } peers { permanent_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } } peers { permanent_uuid: "b1e50122c73046138f5d23aa6860b6e1" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } }
I20260621 02:49:54.303537 32490 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0129aa3f60f411e9d9378703d8c04d6. Latest consensus state: current_term: 1 leader_uuid: "b0129aa3f60f411e9d9378703d8c04d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0129aa3f60f411e9d9378703d8c04d6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } } peers { permanent_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } } peers { permanent_uuid: "b1e50122c73046138f5d23aa6860b6e1" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } } }
I20260621 02:49:54.304085 32490 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:54.308226 32493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:54.310756 32389 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } attempt: 5
W20260621 02:49:54.311673 32417 tablet.cc:2369] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 21 02:49:54 dist-test-slave-hvgn krb5kdc[32235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010193, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.61@KRBTEST.COM for kudu/127.9.166.60@KRBTEST.COM
W20260621 02:49:54.336077 32269 tablet.cc:2369] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:49:54.344370 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:54.311419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:40153 (local address 127.0.0.1:46420)
0621 02:49:54.311872 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:54.311901 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:54.312031 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:54.313326 (+ 1295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:54.313332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:54.313972 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:54.314396 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:54.314416 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.316855 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.316864 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:54.318783 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:54.318803 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.320422 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.320432 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:54.320740 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:54.323572 (+ 2832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:54.323606 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:54.331041 (+ 7435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:54.338870 (+ 7829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:54.338876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:54.338880 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:54.339493 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:54.341309 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:54.341319 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:54.341324 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:54.341520 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:54.342995 (+ 1475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:54.343002 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:54.343200 (+ 198us) client_negotiation.cc:769] Sending connection context
0621 02:49:54.343370 (+ 170us) client_negotiation.cc:240] Negotiation successful
0621 02:49:54.343808 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313}
I20260621 02:49:54.345696 32466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:54.312574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46420 (local address 127.9.166.60:40153)
0621 02:49:54.312738 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.312744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.312766 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:49:54.312834 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.312842 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.312966 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.313078 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.315069 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.316635 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.319025 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.319936 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.320118 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:54.333318 (+ 13200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:54.333325 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:54.333330 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:54.333374 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:54.338706 (+ 5332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.339645 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.339650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.339654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.339818 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.341666 (+ 1848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.341671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.341674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.342018 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:54.342170 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:54.344803 (+ 2633us) server_negotiation.cc:299] Negotiation successful
0621 02:49:54.345108 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:49:54.356662 32319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:54.313282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51760 (local address 127.9.166.62:37277)
0621 02:49:54.313472 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.313478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.326042 (+ 12564us) server_negotiation.cc:407] Connection header received
0621 02:49:54.327415 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.327424 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.327526 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.327612 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.332924 (+ 5312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.334127 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.338512 (+ 4385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.339229 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.339333 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:54.347345 (+ 8012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:54.347351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:54.347356 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:54.347418 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:54.351440 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.352968 (+ 1528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.352974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.352978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.353183 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.353975 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.353980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.353983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.354323 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:54.354583 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:54.355861 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0621 02:49:54.356172 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:49:54.359179 32493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:54.369217 32389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60b8aa64436a4033a7cea3dbcfbb7bf6: Bootstrap starting.
I20260621 02:49:54.374104 32389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60b8aa64436a4033a7cea3dbcfbb7bf6: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:54.375772 32389 log.cc:824] T 00000000000000000000000000000000 P 60b8aa64436a4033a7cea3dbcfbb7bf6: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:54.376142 32506 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260621 02:49:54.383910 32394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:54.321693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51982 (local address 127.9.166.61:41743)
0621 02:49:54.321860 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.321866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.358035 (+ 36169us) server_negotiation.cc:407] Connection header received
0621 02:49:54.358394 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.358399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.358475 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.358546 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.363981 (+ 5435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.365132 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.368785 (+ 3653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.369468 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.369576 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:54.376736 (+ 7160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:54.376745 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:54.376751 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:54.376797 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:54.380352 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.381335 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.381340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.381343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.381519 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.382099 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.382104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.382107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.382455 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:54.382699 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:54.383282 (+ 583us) server_negotiation.cc:299] Negotiation successful
0621 02:49:54.383557 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260621 02:49:54.389529 32389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60b8aa64436a4033a7cea3dbcfbb7bf6: No bootstrap required, opened a new log
I20260621 02:49:54.386230 32248 master_runner.cc:186] Error getting master registration for 127.9.166.61:41743: OK, instance_id { permanent_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6" instance_seqno: 1782010193579379 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 02:49:54.398731 32248 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 02:49:54.404630 32438 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1 [term 1 FOLLOWER]: Refusing update from remote peer b0129aa3f60f411e9d9378703d8c04d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:49:54.405666 32483 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e50122c73046138f5d23aa6860b6e1" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:49:54.418098 32389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60b8aa64436a4033a7cea3dbcfbb7bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0129aa3f60f411e9d9378703d8c04d6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } } peers { permanent_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } } peers { permanent_uuid: "b1e50122c73046138f5d23aa6860b6e1" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } }
I20260621 02:49:54.419000 32389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60b8aa64436a4033a7cea3dbcfbb7bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b8aa64436a4033a7cea3dbcfbb7bf6, State: Initialized, Role: FOLLOWER
I20260621 02:49:54.420018 32389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60b8aa64436a4033a7cea3dbcfbb7bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0129aa3f60f411e9d9378703d8c04d6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } } peers { permanent_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } } peers { permanent_uuid: "b1e50122c73046138f5d23aa6860b6e1" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } }
I20260621 02:49:54.422609 32466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:54.321498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.9.166.60:40153)
0621 02:49:54.345944 (+ 24446us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.345951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.361605 (+ 15654us) server_negotiation.cc:407] Connection header received
0621 02:49:54.361680 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.361688 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.361786 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.361893 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.381914 (+ 20021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.382993 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.385293 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:54.385942 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:54.386038 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:54.412719 (+ 26681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:54.412729 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:54.412736 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:54.412785 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:54.416738 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.417718 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.417724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.417728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.417871 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:54.419632 (+ 1761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:54.419637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:54.419641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:54.420081 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:54.420374 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:54.421871 (+ 1497us) server_negotiation.cc:299] Negotiation successful
0621 02:49:54.422138 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24323,"thread_start_us":41439,"threads_started":1}
I20260621 02:49:54.428596 32469 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0129aa3f60f411e9d9378703d8c04d6. Latest consensus state: current_term: 1 leader_uuid: "b0129aa3f60f411e9d9378703d8c04d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0129aa3f60f411e9d9378703d8c04d6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } } peers { permanent_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } } peers { permanent_uuid: "b1e50122c73046138f5d23aa6860b6e1" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } } }
I20260621 02:49:54.428936 32469 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:49:54.431964 32511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60b8aa64436a4033a7cea3dbcfbb7bf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0129aa3f60f411e9d9378703d8c04d6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } } peers { permanent_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } } peers { permanent_uuid: "b1e50122c73046138f5d23aa6860b6e1" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } } }
I20260621 02:49:54.432407 32511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60b8aa64436a4033a7cea3dbcfbb7bf6 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:49:54.434912 32508 mvcc.cc:205] Tried to move back new op lower bound from 7299113756267802624 to 7299113755851010048. Current Snapshot: MvccSnapshot[applied={T|T < 7299113756267802624}]
I20260621 02:49:54.436056 32364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b0129aa3f60f411e9d9378703d8c04d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6"
I20260621 02:49:54.436520 32364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60b8aa64436a4033a7cea3dbcfbb7bf6 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:54.440090 32509 mvcc.cc:205] Tried to move back new op lower bound from 7299113756267802624 to 7299113755851010048. Current Snapshot: MvccSnapshot[applied={T|T < 7299113756267802624}]
I20260621 02:49:54.441793 32364 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60b8aa64436a4033a7cea3dbcfbb7bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0129aa3f60f411e9d9378703d8c04d6 in term 1.
I20260621 02:49:54.442672 32363 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60b8aa64436a4033a7cea3dbcfbb7bf6 [term 1 FOLLOWER]: Refusing update from remote peer b0129aa3f60f411e9d9378703d8c04d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:49:54.444492 32490 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:49:54.447019 32483 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0129aa3f60f411e9d9378703d8c04d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0129aa3f60f411e9d9378703d8c04d6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } } peers { permanent_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } } peers { permanent_uuid: "b1e50122c73046138f5d23aa6860b6e1" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } } }
I20260621 02:49:54.447315 32483 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:54.448055 32493 catalog_manager.cc:1347] Generated new cluster ID: 1d748d1cc0154d55b82bd74a6beaeb58
I20260621 02:49:54.448140 32493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:54.451855 32389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60b8aa64436a4033a7cea3dbcfbb7bf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:54.466547 32469 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0129aa3f60f411e9d9378703d8c04d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0129aa3f60f411e9d9378703d8c04d6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } } peers { permanent_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } } peers { permanent_uuid: "b1e50122c73046138f5d23aa6860b6e1" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } } }
I20260621 02:49:54.466879 32469 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:49:54.473915 32493 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:54.475797 32493 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:54.480443 32517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60b8aa64436a4033a7cea3dbcfbb7bf6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0129aa3f60f411e9d9378703d8c04d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0129aa3f60f411e9d9378703d8c04d6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } } peers { permanent_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } } peers { permanent_uuid: "b1e50122c73046138f5d23aa6860b6e1" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } } }
I20260621 02:49:54.480866 32517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60b8aa64436a4033a7cea3dbcfbb7bf6 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:49:54.481248 32511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60b8aa64436a4033a7cea3dbcfbb7bf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0129aa3f60f411e9d9378703d8c04d6. Latest consensus state: current_term: 1 leader_uuid: "b0129aa3f60f411e9d9378703d8c04d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0129aa3f60f411e9d9378703d8c04d6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } } peers { permanent_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } } peers { permanent_uuid: "b1e50122c73046138f5d23aa6860b6e1" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } } }
I20260621 02:49:54.481480 32511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60b8aa64436a4033a7cea3dbcfbb7bf6 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:49:54.482667 32483 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0129aa3f60f411e9d9378703d8c04d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0129aa3f60f411e9d9378703d8c04d6" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 37277 } } peers { permanent_uuid: "60b8aa64436a4033a7cea3dbcfbb7bf6" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 41743 } } peers { permanent_uuid: "b1e50122c73046138f5d23aa6860b6e1" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 40153 } } }
I20260621 02:49:54.482932 32483 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6 [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:54.493992 32493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0129aa3f60f411e9d9378703d8c04d6: Generated new TSK 0
I20260621 02:49:54.495028 32493 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260621 02:49:54.530493 32530 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260621 02:49:54.599344 32319 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:54.574833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51768 (local address 127.9.166.62:37277)
0621 02:49:54.575226 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.575232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.597207 (+ 21975us) server_negotiation.cc:407] Connection header received
0621 02:49:54.597685 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.597692 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.597781 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.597854 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.599108 (+ 1254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51768: BlockingRecv error: recv got EOF from 127.0.0.1:51768 (error 108)
Metrics: {"server-negotiator.queue_time_us":248}
W20260621 02:49:54.601609 32394 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:54.573739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51984 (local address 127.9.166.61:41743)
0621 02:49:54.574590 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.574597 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.600115 (+ 25518us) server_negotiation.cc:407] Connection header received
0621 02:49:54.600379 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.600384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.600470 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.600540 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.601412 (+ 872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51984: BlockingRecv error: recv got EOF from 127.0.0.1:51984 (error 108)
Metrics: {"server-negotiator.queue_time_us":659}
W20260621 02:49:54.603896 32466 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:54.575013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46448 (local address 127.9.166.60:40153)
0621 02:49:54.575167 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.575173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.602318 (+ 27145us) server_negotiation.cc:407] Connection header received
0621 02:49:54.602572 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.602580 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.602668 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.602737 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.603663 (+ 926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46448: BlockingRecv error: recv got EOF from 127.0.0.1:46448 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260621 02:49:54.638051 32394 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:54.630322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51998 (local address 127.9.166.61:41743)
0621 02:49:54.630510 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.630518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.635990 (+ 5472us) server_negotiation.cc:407] Connection header received
0621 02:49:54.636247 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.636255 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.636384 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.636508 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.637785 (+ 1277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51998: BlockingRecv error: recv got EOF from 127.0.0.1:51998 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260621 02:49:54.645041 32394 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:54.643019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52008 (local address 127.9.166.61:41743)
0621 02:49:54.643233 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.643241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.643722 (+ 481us) server_negotiation.cc:407] Connection header received
0621 02:49:54.643930 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.643936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.644011 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.644114 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.644837 (+ 723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52008: BlockingRecv error: recv got EOF from 127.0.0.1:52008 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260621 02:49:54.652037 32394 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:54.649948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52016 (local address 127.9.166.61:41743)
0621 02:49:54.650163 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.650171 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.650658 (+ 487us) server_negotiation.cc:407] Connection header received
0621 02:49:54.650870 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.650876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.650965 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.651072 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.651798 (+ 726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52016: BlockingRecv error: recv got EOF from 127.0.0.1:52016 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260621 02:49:54.660881 32394 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:54.658803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52022 (local address 127.9.166.61:41743)
0621 02:49:54.659058 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.659067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.659581 (+ 514us) server_negotiation.cc:407] Connection header received
0621 02:49:54.659783 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.659788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.659868 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.659972 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.660653 (+ 681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52022: BlockingRecv error: recv got EOF from 127.0.0.1:52022 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260621 02:49:54.673563 32394 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:54.671520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.9.166.61:41743)
0621 02:49:54.671724 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.671732 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.672244 (+ 512us) server_negotiation.cc:407] Connection header received
0621 02:49:54.672455 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.672461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.672537 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.672635 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.673325 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52032: BlockingRecv error: recv got EOF from 127.0.0.1:52032 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260621 02:49:54.695443 32394 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:54.693291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52048 (local address 127.9.166.61:41743)
0621 02:49:54.693517 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.693527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.694064 (+ 537us) server_negotiation.cc:407] Connection header received
0621 02:49:54.694288 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.694294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.694362 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.694465 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.695195 (+ 730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52048: BlockingRecv error: recv got EOF from 127.0.0.1:52048 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260621 02:49:54.733144 32394 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:54.730711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52064 (local address 127.9.166.61:41743)
0621 02:49:54.730934 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.730943 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.731552 (+ 609us) server_negotiation.cc:407] Connection header received
0621 02:49:54.731798 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.731802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.731874 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.731981 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.732882 (+ 901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52064: BlockingRecv error: recv got EOF from 127.0.0.1:52064 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260621 02:49:54.802949 32394 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:54.800937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52076 (local address 127.9.166.61:41743)
0621 02:49:54.801169 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.801176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.801497 (+ 321us) server_negotiation.cc:407] Connection header received
0621 02:49:54.801752 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.801757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.801832 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.801947 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.802716 (+ 769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52076: BlockingRecv error: recv got EOF from 127.0.0.1:52076 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260621 02:49:54.936368 32394 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:54.934366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52090 (local address 127.9.166.61:41743)
0621 02:49:54.934583 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:54.934591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:54.934981 (+ 390us) server_negotiation.cc:407] Connection header received
0621 02:49:54.935206 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:54.935211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:54.935289 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:54.935395 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:54.936161 (+ 766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52090: BlockingRecv error: recv got EOF from 127.0.0.1:52090 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:49:55.178948 32480 catalog_manager.cc:1259] Loaded cluster ID: 1d748d1cc0154d55b82bd74a6beaeb58
I20260621 02:49:55.179052 32480 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1: loading cluster ID for follower catalog manager: success
I20260621 02:49:55.181578 32480 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1: acquiring CA information for follower catalog manager: success
I20260621 02:49:55.183303 32480 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b1e50122c73046138f5d23aa6860b6e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 02:49:55.197669 32394 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:55.196098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.9.166.61:41743)
0621 02:49:55.196284 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:55.196288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:55.196556 (+ 268us) server_negotiation.cc:407] Connection header received
0621 02:49:55.196721 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:55.196725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:55.196805 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:55.196900 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:55.197509 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52098: BlockingRecv error: recv got EOF from 127.0.0.1:52098 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260621 02:49:55.534788 32530 catalog_manager.cc:1259] Loaded cluster ID: 1d748d1cc0154d55b82bd74a6beaeb58
I20260621 02:49:55.534905 32530 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60b8aa64436a4033a7cea3dbcfbb7bf6: loading cluster ID for follower catalog manager: success
I20260621 02:49:55.537582 32530 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60b8aa64436a4033a7cea3dbcfbb7bf6: acquiring CA information for follower catalog manager: success
I20260621 02:49:55.539407 32530 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60b8aa64436a4033a7cea3dbcfbb7bf6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 02:49:55.723861 32596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:55.716852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52112 (local address 127.9.166.61:41743)
0621 02:49:55.717715 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:55.717721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:55.717743 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:49:55.717829 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:55.717834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:55.718013 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:55.718146 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:49:55.718557 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:55.719797 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:55.721138 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:55.721854 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:55.721995 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:55.722446 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:49:55.723161 (+ 715us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:49:55.723311 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:55.723377 (+ 66us) server_negotiation.cc:299] Negotiation successful
0621 02:49:55.723493 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":545,"threads_started":1}
I20260621 02:49:55.736295 32602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:55.730473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.9.166.60:40153)
0621 02:49:55.731127 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:55.731135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:55.731155 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:49:55.731264 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:55.731268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:55.731458 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:55.731637 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:49:55.732022 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:55.732744 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:55.734044 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:55.734536 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:55.734675 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:55.735156 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:49:55.735701 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:49:55.735873 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:55.735934 (+ 61us) server_negotiation.cc:299] Negotiation successful
0621 02:49:55.736016 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":417,"threads_started":1}
I20260621 02:49:55.760196 32610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:55.744050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51770 (local address 127.9.166.62:37277)
0621 02:49:55.752215 (+ 8165us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:55.752221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:55.752243 (+ 22us) server_negotiation.cc:407] Connection header received
0621 02:49:55.752358 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:55.752365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:55.752570 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:55.753428 (+ 858us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:49:55.753592 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:55.754653 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:55.757113 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:55.757775 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:55.758360 (+ 585us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:55.758556 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:49:55.759009 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:49:55.759705 (+ 696us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:55.759771 (+ 66us) server_negotiation.cc:299] Negotiation successful
0621 02:49:55.759869 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8036,"thread_start_us":7924,"threads_started":1}
I20260621 02:49:55.760807 32602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:55.743811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46458 (local address 127.9.166.60:40153)
0621 02:49:55.744362 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:55.744371 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:55.751727 (+ 7356us) server_negotiation.cc:407] Connection header received
0621 02:49:55.751883 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:55.751894 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:55.752028 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:55.752141 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:49:55.753106 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:55.754517 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:55.758566 (+ 4049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:55.759143 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:55.759248 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:55.759911 (+ 663us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:49:55.760029 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:49:55.760094 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:55.760453 (+ 359us) server_negotiation.cc:299] Negotiation successful
0621 02:49:55.760535 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399}
I20260621 02:49:55.764196 32596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:55.743526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52114 (local address 127.9.166.61:41743)
0621 02:49:55.743714 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:55.743720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:55.756265 (+ 12545us) server_negotiation.cc:407] Connection header received
0621 02:49:55.756413 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:55.756418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:55.756502 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:55.756610 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:49:55.757359 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:55.758434 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:55.761780 (+ 3346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:55.762397 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:55.762487 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:55.763284 (+ 797us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:49:55.763432 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:49:55.763534 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:55.763761 (+ 227us) server_negotiation.cc:299] Negotiation successful
0621 02:49:55.763845 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260621 02:49:55.765178 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 32248
I20260621 02:49:55.766208 32310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:55.883265 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 32248
I20260621 02:49:55.904501 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 32322
I20260621 02:49:55.905668 32384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:56.013054 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 32322
I20260621 02:49:56.033619 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 32396
I20260621 02:49:56.034827 32458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:49:56.140995 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 32396
2026-06-21T02:49:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4819 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 21 02:49:56 dist-test-slave-hvgn krb5kdc[32616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 21 02:49:56 dist-test-slave-hvgn krb5kdc[32616](info): set up 2 sockets
Jun 21 02:49:56 dist-test-slave-hvgn krb5kdc[32616](info): commencing operation
krb5kdc: starting...
W20260621 02:49:57.269464 9880 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/test-user.keytab.
Jun 21 02:49:57 dist-test-slave-hvgn krb5kdc[32616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-21T02:49:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-21T02:49:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.166.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
Entry for principal kudu/127.9.166.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab.
I20260621 02:49:57.511092 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:36703
--webserver_interface=127.9.166.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:35717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:36703,127.9.166.61:39585,127.9.166.60:37189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260621 02:49:57.836649 32629 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 02:49:57.836951 32629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:57.837021 32629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:57.837108 32629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:57.844430 32629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:57.844514 32629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:57.844568 32629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:57.844617 32629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:57.844676 32629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:57.854362 32629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:35717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:36703,127.9.166.61:39585,127.9.166.60:37189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.62.keytab
--principal=kudu/127.9.166.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.62:36703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.166.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:57.855573 32629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:57.856815 32629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:57.868772 32637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:57.868849 32634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:57.868867 32635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:57.869763 32629 server_base.cc:1034] running on GCE node
I20260621 02:49:57.870532 32629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:57.871860 32629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:57.873170 32629 hybrid_clock.cc:648] HybridClock initialized: now 1782010197873066 us; error 41 us; skew 500 ppm
I20260621 02:49:57.873538 32629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:57 dist-test-slave-hvgn krb5kdc[32616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010197, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:57.876767 32629 init.cc:377] Logged in from keytab as kudu/127.9.166.62@KRBTEST.COM (short username kudu)
I20260621 02:49:57.878237 32629 webserver.cc:460] Webserver started at http://127.9.166.62:39023/ using document root <none> and password file <none>
I20260621 02:49:57.878908 32629 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:57.879037 32629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:57.879422 32629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:57.882305 32629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-0/data/instance:
uuid: "161c1bf37b924dfab17cac785abf5dab"
format_stamp: "Formatted at 2026-06-21 02:49:57 on dist-test-slave-hvgn"
server_key: "44a1759e7344cd185cab8b983926ef68"
server_key_iv: "c58d5a783ec2dd0553f52452b2248b1d"
server_key_version: "encryptionkey@0"
I20260621 02:49:57.883070 32629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-0/wal/instance:
uuid: "161c1bf37b924dfab17cac785abf5dab"
format_stamp: "Formatted at 2026-06-21 02:49:57 on dist-test-slave-hvgn"
server_key: "44a1759e7344cd185cab8b983926ef68"
server_key_iv: "c58d5a783ec2dd0553f52452b2248b1d"
server_key_version: "encryptionkey@0"
I20260621 02:49:57.887972 32629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260621 02:49:57.891280 32644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:57.892596 32629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260621 02:49:57.892771 32629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-0/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-0/wal
uuid: "161c1bf37b924dfab17cac785abf5dab"
format_stamp: "Formatted at 2026-06-21 02:49:57 on dist-test-slave-hvgn"
server_key: "44a1759e7344cd185cab8b983926ef68"
server_key_iv: "c58d5a783ec2dd0553f52452b2248b1d"
server_key_version: "encryptionkey@0"
I20260621 02:49:57.892985 32629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:57.912402 32629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:57.916239 32629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:57.916507 32629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:57.935104 32629 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.62:36703
I20260621 02:49:57.935115 32695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.62:36703 every 8 connection(s)
I20260621 02:49:57.936527 32629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-0/data/info.pb
I20260621 02:49:57.940131 32696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:57.943750 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 32629
I20260621 02:49:57.943935 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-0/wal/instance
I20260621 02:49:57.944463 9880 external_mini_cluster.cc:1428] Setting key 6e8b5fb4596ee7327681a1b2130cc542
I20260621 02:49:57.948439 32696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:49:57 dist-test-slave-hvgn krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010197, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.166.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.61@KRBTEST.COM" created.
I20260621 02:49:57.974066 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:57.952354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:36703 (local address 127.0.0.1:51388)
0621 02:49:57.953631 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:57.953679 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:57.955751 (+ 2072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:57.956752 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:57.956757 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:57.957353 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:57.957685 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:57.957706 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:57.960055 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:57.960060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:57.961179 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:57.961191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:57.962505 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:57.962516 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:57.962669 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:57.963331 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:57.963354 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:57.965555 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:57.969907 (+ 4352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:57.969917 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:57.969925 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:57.970325 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:57.970839 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:57.970842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:57.970845 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:57.971014 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:57.972000 (+ 986us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:57.972006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:57.972208 (+ 202us) client_negotiation.cc:769] Sending connection context
0621 02:49:57.972430 (+ 222us) client_negotiation.cc:240] Negotiation successful
0621 02:49:57.972879 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":4,"thread_start_us":824,"threads_started":1}
I20260621 02:49:57.974040 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:57.952354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51388 (local address 127.9.166.62:36703)
0621 02:49:57.953230 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:57.953251 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:57.953813 (+ 562us) server_negotiation.cc:407] Connection header received
0621 02:49:57.956188 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:57.956199 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:57.956494 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:57.956693 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:57.957916 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:57.959886 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:57.961324 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:57.962278 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:57.962455 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:57.965797 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:57.965809 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:57.965827 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:57.965867 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:57.969772 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:57.970477 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:57.970487 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:57.970493 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:57.970673 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:57.971139 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:57.971147 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:57.971152 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:57.971857 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:57.972073 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:57.972609 (+ 536us) server_negotiation.cc:299] Negotiation successful
0621 02:49:57.972983 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":190,"server-negotiator.queue_time_us":626,"thread_start_us":400,"threads_started":1}
I20260621 02:49:57.977047 32696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:49:57.978037 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:57.977677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:39585 (local address 127.0.0.1:43230)
0621 02:49:57.977833 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:57.977930 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260621 02:49:57.978282 32646 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.166.61:39585: connect: Connection refused (error 111)
I20260621 02:49:57.979813 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:57.979477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:39585 (local address 127.0.0.1:43232)
0621 02:49:57.979632 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:57.979698 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260621 02:49:57.980032 32696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.61:39585: Network error: Client connection negotiation failed: client connection to 127.9.166.61:39585: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.61.keytab.
Entry for principal kudu/127.9.166.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.61.keytab.
I20260621 02:49:58.002732 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.61:39585
--webserver_interface=127.9.166.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.61.keytab
--principal=kudu/127.9.166.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:35717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:36703,127.9.166.61:39585,127.9.166.60:37189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:49:58.029346 32696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } attempt: 1
I20260621 02:49:58.030601 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.030139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:39585 (local address 127.0.0.1:43242)
0621 02:49:58.030373 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.030472 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260621 02:49:58.031692 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.031285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:39585 (local address 127.0.0.1:43258)
0621 02:49:58.031505 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.031570 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260621 02:49:58.031919 32696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.61:39585: Network error: Client connection negotiation failed: client connection to 127.9.166.61:39585: connect: Connection refused (error 111)
I20260621 02:49:58.100203 32696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } attempt: 2
I20260621 02:49:58.101380 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.100898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:39585 (local address 127.0.0.1:43266)
0621 02:49:58.101126 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.101243 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260621 02:49:58.102432 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.102063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:39585 (local address 127.0.0.1:43278)
0621 02:49:58.102236 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.102302 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260621 02:49:58.102680 32696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.61:39585: Network error: Client connection negotiation failed: client connection to 127.9.166.61:39585: connect: Connection refused (error 111)
I20260621 02:49:58.193957 32696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } attempt: 3
I20260621 02:49:58.195245 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.194776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:39585 (local address 127.0.0.1:43280)
0621 02:49:58.194996 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.195109 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260621 02:49:58.196293 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.195933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:39585 (local address 127.0.0.1:43284)
0621 02:49:58.196085 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.196163 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260621 02:49:58.196602 32696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.61:39585: Network error: Client connection negotiation failed: client connection to 127.9.166.61:39585: connect: Connection refused (error 111)
W20260621 02:49:58.326022 32703 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 02:49:58.326373 32703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:58.326500 32703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:58.326632 32703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:58.333746 32703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:58.333845 32703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:58.333940 32703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:58.334028 32703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:58.334146 32703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:58.339856 32696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } attempt: 4
I20260621 02:49:58.341066 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.340519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:39585 (local address 127.0.0.1:43286)
0621 02:49:58.340842 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.340937 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260621 02:49:58.342118 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.341763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:39585 (local address 127.0.0.1:43288)
0621 02:49:58.341917 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.342004 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.61:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 02:49:58.342316 32696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.61:39585: Network error: Client connection negotiation failed: client connection to 127.9.166.61:39585: connect: Connection refused (error 111)
I20260621 02:49:58.344703 32703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:35717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:36703,127.9.166.61:39585,127.9.166.60:37189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.61.keytab
--principal=kudu/127.9.166.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.61:39585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.166.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:58.346031 32703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:58.347345 32703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260621 02:49:58.358619 32708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:58.358729 32711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:58.358940 32709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:58.359709 32703 server_base.cc:1034] running on GCE node
I20260621 02:49:58.360424 32703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:58.361724 32703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:58.362946 32703 hybrid_clock.cc:648] HybridClock initialized: now 1782010198362888 us; error 23 us; skew 500 ppm
I20260621 02:49:58.363276 32703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:58 dist-test-slave-hvgn krb5kdc[32616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010198, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:58.366495 32703 init.cc:377] Logged in from keytab as kudu/127.9.166.61@KRBTEST.COM (short username kudu)
I20260621 02:49:58.367964 32703 webserver.cc:460] Webserver started at http://127.9.166.61:42963/ using document root <none> and password file <none>
I20260621 02:49:58.368628 32703 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:58.368742 32703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:58.369061 32703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:58.371953 32703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-1/data/instance:
uuid: "688b96cbba634b73a853dc26ca79dd0c"
format_stamp: "Formatted at 2026-06-21 02:49:58 on dist-test-slave-hvgn"
server_key: "ac3b01fb726e8b210e17afa7f93ce958"
server_key_iv: "049bdbcd3e261edd30c2b273fd59c379"
server_key_version: "encryptionkey@0"
I20260621 02:49:58.372839 32703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-1/wal/instance:
uuid: "688b96cbba634b73a853dc26ca79dd0c"
format_stamp: "Formatted at 2026-06-21 02:49:58 on dist-test-slave-hvgn"
server_key: "ac3b01fb726e8b210e17afa7f93ce958"
server_key_iv: "049bdbcd3e261edd30c2b273fd59c379"
server_key_version: "encryptionkey@0"
I20260621 02:49:58.377676 32703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260621 02:49:58.380888 32718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:58.382226 32703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260621 02:49:58.382397 32703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-1/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-1/wal
uuid: "688b96cbba634b73a853dc26ca79dd0c"
format_stamp: "Formatted at 2026-06-21 02:49:58 on dist-test-slave-hvgn"
server_key: "ac3b01fb726e8b210e17afa7f93ce958"
server_key_iv: "049bdbcd3e261edd30c2b273fd59c379"
server_key_version: "encryptionkey@0"
I20260621 02:49:58.382645 32703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:58.400323 32703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:58.404546 32703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:58.404814 32703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:58.423074 32703 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.61:39585
I20260621 02:49:58.423094 301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.61:39585 every 8 connection(s)
I20260621 02:49:58.424440 32703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-1/data/info.pb
I20260621 02:49:58.426782 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 32703
I20260621 02:49:58.426975 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-1/wal/instance
I20260621 02:49:58.427565 9880 external_mini_cluster.cc:1428] Setting key 86112bd15844a10b243d858dd316c372
I20260621 02:49:58.428776 302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:58.438566 302 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:49:58 dist-test-slave-hvgn krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010198, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.61@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.166.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.166.60@KRBTEST.COM" created.
I20260621 02:49:58.458734 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.441220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.9.166.62:36703)
0621 02:49:58.441418 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:58.441423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:58.442703 (+ 1280us) server_negotiation.cc:407] Connection header received
0621 02:49:58.444649 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:58.444654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:58.444744 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:58.444854 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:58.446359 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.447057 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.448652 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.449063 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.449201 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:58.452690 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:58.452695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:58.452698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:58.452732 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:58.455727 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:58.456449 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:58.456454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:58.456458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:58.456582 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:58.457005 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:58.457010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:58.457013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:58.457304 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:58.457460 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:58.458181 (+ 721us) server_negotiation.cc:299] Negotiation successful
0621 02:49:58.458411 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260621 02:49:58.459821 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.441736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:36703 (local address 127.0.0.1:51394)
0621 02:49:58.442532 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.442568 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:58.444358 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:58.445083 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:58.445091 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:58.445635 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:58.446189 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:58.446207 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.447233 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.447237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:58.448489 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:58.448501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.449312 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.449321 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:58.449546 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:58.450228 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:58.450251 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:58.452455 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:58.455866 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:58.455875 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:58.455900 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:58.456318 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:58.456701 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:58.456706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:58.456710 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:58.456883 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:58.457439 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:58.457447 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:58.457790 (+ 343us) client_negotiation.cc:769] Sending connection context
0621 02:49:58.458051 (+ 261us) client_negotiation.cc:240] Negotiation successful
0621 02:49:58.458582 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":379,"threads_started":1}
I20260621 02:49:58.461535 302 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:49:58 dist-test-slave-hvgn krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010198, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.61@KRBTEST.COM for kudu/127.9.166.61@KRBTEST.COM
I20260621 02:49:58.477308 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.462347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:39585 (local address 127.0.0.1:43292)
0621 02:49:58.462511 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.462526 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:58.462601 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:58.463900 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:58.463904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:58.464308 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:58.464509 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:58.464523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.465869 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.465877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:58.467446 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:58.467463 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.468310 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.468318 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:58.468534 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:58.469128 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:58.469154 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:58.471656 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:58.474824 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:58.474832 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:58.474837 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:58.475190 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:58.475627 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:58.475632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:58.475635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:58.475772 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:58.476351 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:58.476356 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:58.476506 (+ 150us) client_negotiation.cc:769] Sending connection context
0621 02:49:58.476664 (+ 158us) client_negotiation.cc:240] Negotiation successful
0621 02:49:58.476938 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260621 02:49:58.477449 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.462619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43292 (local address 127.9.166.61:39585)
0621 02:49:58.463338 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:58.463347 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:58.463379 (+ 32us) server_negotiation.cc:407] Connection header received
0621 02:49:58.463470 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:58.463478 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:58.463726 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:58.463884 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:58.464681 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.465732 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.467602 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.468117 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.468252 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:58.471854 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:58.471864 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:58.471875 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:58.471901 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:58.474681 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:58.475317 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:58.475333 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:58.475336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:58.475511 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:58.475874 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:58.475881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:58.475884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:58.476230 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:58.476427 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:58.476808 (+ 381us) server_negotiation.cc:299] Negotiation successful
0621 02:49:58.477052 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":495,"thread_start_us":320,"threads_started":1}
I20260621 02:49:58.480191 302 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:49:58.481341 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.480827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54242)
0621 02:49:58.481046 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.481184 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260621 02:49:58.481590 32722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
I20260621 02:49:58.483012 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.482654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54258)
0621 02:49:58.482791 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.482900 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260621 02:49:58.483247 302 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:37189: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.166.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.60.keytab.
Entry for principal kudu/127.9.166.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.60.keytab.
I20260621 02:49:58.486534 9880 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2e3lzq/build/asan/bin/kudu
/tmp/dist-test-task2e3lzq/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.60:37189
--webserver_interface=127.9.166.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.60.keytab
--principal=kudu/127.9.166.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.166.20:35717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.166.62:36703,127.9.166.61:39585,127.9.166.60:37189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260621 02:49:58.532598 302 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } attempt: 1
I20260621 02:49:58.533936 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.533430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54268)
0621 02:49:58.533665 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.533773 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260621 02:49:58.535043 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.534691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54272)
0621 02:49:58.534841 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.534941 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260621 02:49:58.535254 302 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:37189: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
I20260621 02:49:58.603482 302 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } attempt: 2
I20260621 02:49:58.604663 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.604186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54286)
0621 02:49:58.604391 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.604489 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260621 02:49:58.605639 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.605308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54292)
0621 02:49:58.605453 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.605528 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260621 02:49:58.605839 302 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:37189: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
I20260621 02:49:58.641597 32696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } attempt: 5
Jun 21 02:49:58 dist-test-slave-hvgn krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010197, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for kudu/127.9.166.61@KRBTEST.COM
I20260621 02:49:58.656699 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.642398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:39585 (local address 127.0.0.1:43296)
0621 02:49:58.642636 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.642659 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:58.642799 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:58.643154 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:58.643159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:58.643715 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:58.643964 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:58.643980 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.644981 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.644989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:58.646679 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:58.646699 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.647466 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.647474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:58.647734 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:58.648481 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:58.648510 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:58.651200 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:58.654259 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:58.654265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:58.654269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:58.654627 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:58.655006 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:58.655011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:58.655015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:58.655154 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:58.655738 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:58.655743 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:58.655901 (+ 158us) client_negotiation.cc:769] Sending connection context
0621 02:49:58.656045 (+ 144us) client_negotiation.cc:240] Negotiation successful
0621 02:49:58.656318 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260621 02:49:58.656791 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.642512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.9.166.61:39585)
0621 02:49:58.642703 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:58.642707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:58.642727 (+ 20us) server_negotiation.cc:407] Connection header received
0621 02:49:58.642929 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:58.642933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:58.642994 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:58.643078 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:58.644130 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.644816 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.646838 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.647271 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.647368 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:58.651459 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:58.651465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:58.651468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:58.651500 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:58.654113 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:58.654762 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:58.654769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:58.654773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:58.654896 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:58.655270 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:58.655275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:58.655278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:58.655618 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:58.655776 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:58.656151 (+ 375us) server_negotiation.cc:299] Negotiation successful
0621 02:49:58.656369 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260621 02:49:58.657696 32696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:49:58.658867 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.658328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54306)
0621 02:49:58.658569 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.658686 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260621 02:49:58.659914 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.659519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54318)
0621 02:49:58.659697 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.659794 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260621 02:49:58.660117 32696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:37189: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
I20260621 02:49:58.697050 302 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } attempt: 3
I20260621 02:49:58.698179 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.697693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54326)
0621 02:49:58.697925 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.698020 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260621 02:49:58.699319 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.698910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54340)
0621 02:49:58.699139 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.699217 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260621 02:49:58.699545 302 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:37189: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
I20260621 02:49:58.711364 32696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } attempt: 1
I20260621 02:49:58.712731 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.712164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54342)
0621 02:49:58.712402 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.712513 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260621 02:49:58.713820 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.713380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54344)
0621 02:49:58.713557 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.713664 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260621 02:49:58.714031 32696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:37189: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
I20260621 02:49:58.782274 32696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } attempt: 2
I20260621 02:49:58.783622 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.783015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54354)
0621 02:49:58.783267 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.783412 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260621 02:49:58.784807 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.784385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54356)
0621 02:49:58.784555 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.784659 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260621 02:49:58.785044 32696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:37189: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
W20260621 02:49:58.820771 309 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260621 02:49:58.821081 309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260621 02:49:58.821161 309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260621 02:49:58.821244 309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260621 02:49:58.828359 309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260621 02:49:58.828454 309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260621 02:49:58.828510 309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260621 02:49:58.828560 309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260621 02:49:58.828621 309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260621 02:49:58.840044 309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.166.20:35717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.166.62:36703,127.9.166.61:39585,127.9.166.60:37189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/krb5kdc/kudu_127.9.166.60.keytab
--principal=kudu/127.9.166.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.166.60:37189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.166.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 21 Jun 2026 01:47:56 UTC on ccycloud.quasar-luwggy.root.comops.site
build id 4457866
ASAN enabled
I20260621 02:49:58.841356 309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260621 02:49:58.842718 309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260621 02:49:58.842764 302 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } attempt: 4
I20260621 02:49:58.843982 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.843521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54372)
0621 02:49:58.843736 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.843828 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260621 02:49:58.844974 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.844635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54374)
0621 02:49:58.844825 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.844886 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260621 02:49:58.845253 302 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:37189: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
W20260621 02:49:58.854979 314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:58.855039 315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260621 02:49:58.855000 317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260621 02:49:58.855958 309 server_base.cc:1034] running on GCE node
I20260621 02:49:58.856686 309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260621 02:49:58.858214 309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260621 02:49:58.859501 309 hybrid_clock.cc:648] HybridClock initialized: now 1782010198859409 us; error 51 us; skew 500 ppm
I20260621 02:49:58.859844 309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 21 02:49:58 dist-test-slave-hvgn krb5kdc[32616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010198, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260621 02:49:58.863186 309 init.cc:377] Logged in from keytab as kudu/127.9.166.60@KRBTEST.COM (short username kudu)
I20260621 02:49:58.864950 309 webserver.cc:460] Webserver started at http://127.9.166.60:40929/ using document root <none> and password file <none>
I20260621 02:49:58.865660 309 fs_manager.cc:359] Metadata directory not provided
I20260621 02:49:58.865790 309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260621 02:49:58.866127 309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260621 02:49:58.869122 309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-2/data/instance:
uuid: "c542b39816774df39653aa04b2f26507"
format_stamp: "Formatted at 2026-06-21 02:49:58 on dist-test-slave-hvgn"
server_key: "45ec689f26a52719ce05cce0b0a69994"
server_key_iv: "66b851a0fa0f64285ee5b0f30e4fce15"
server_key_version: "encryptionkey@0"
I20260621 02:49:58.869885 309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-2/wal/instance:
uuid: "c542b39816774df39653aa04b2f26507"
format_stamp: "Formatted at 2026-06-21 02:49:58 on dist-test-slave-hvgn"
server_key: "45ec689f26a52719ce05cce0b0a69994"
server_key_iv: "66b851a0fa0f64285ee5b0f30e4fce15"
server_key_version: "encryptionkey@0"
I20260621 02:49:58.875022 309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260621 02:49:58.878544 324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260621 02:49:58.880021 309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260621 02:49:58.880216 309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-2/data,/tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-2/wal
uuid: "c542b39816774df39653aa04b2f26507"
format_stamp: "Formatted at 2026-06-21 02:49:58 on dist-test-slave-hvgn"
server_key: "45ec689f26a52719ce05cce0b0a69994"
server_key_iv: "66b851a0fa0f64285ee5b0f30e4fce15"
server_key_version: "encryptionkey@0"
I20260621 02:49:58.880439 309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260621 02:49:58.891264 32696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } attempt: 3
I20260621 02:49:58.892392 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.891885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54384)
0621 02:49:58.892113 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.892212 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260621 02:49:58.893473 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.893042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54390)
0621 02:49:58.893275 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.893353 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260621 02:49:58.893719 32696 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.166.60:37189: Network error: Client connection negotiation failed: client connection to 127.9.166.60:37189: connect: Connection refused (error 111)
I20260621 02:49:58.903308 309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260621 02:49:58.907610 309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260621 02:49:58.907867 309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260621 02:49:58.926383 309 rpc_server.cc:307] RPC server started. Bound to: 127.9.166.60:37189
I20260621 02:49:58.926405 375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.166.60:37189 every 8 connection(s)
I20260621 02:49:58.927772 309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-2/data/info.pb
I20260621 02:49:58.931499 376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260621 02:49:58.931660 9880 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2e3lzq/build/asan/bin/kudu as pid 309
I20260621 02:49:58.931864 9880 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2e3lzq/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1782009844354615-9880-0/minicluster-data/master-2/wal/instance
I20260621 02:49:58.932340 9880 external_mini_cluster.cc:1428] Setting key 6fc642b50c8f0d33e42fe6ca9a8cb3be
I20260621 02:49:58.941918 376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:49:58 dist-test-slave-hvgn krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.60@KRBTEST.COM
Jun 21 02:49:58 dist-test-slave-hvgn krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010198, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.60@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:49:58.959579 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.945166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51396 (local address 127.9.166.62:36703)
0621 02:49:58.945416 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:58.945426 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:58.945948 (+ 522us) server_negotiation.cc:407] Connection header received
0621 02:49:58.946180 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:58.946186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:58.946272 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:58.946382 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:58.947493 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.948204 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.949633 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.950408 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.950536 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:58.954033 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:58.954038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:58.954042 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:58.954081 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:58.956838 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:58.957514 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:58.957524 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:58.957527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:58.957666 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:58.958059 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:58.958065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:58.958068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:58.958338 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:58.958496 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:58.958980 (+ 484us) server_negotiation.cc:299] Negotiation successful
0621 02:49:58.959191 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260621 02:49:58.960176 379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.934213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54398 (local address 127.9.166.60:37189)
0621 02:49:58.935191 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:58.935207 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:58.935259 (+ 52us) server_negotiation.cc:407] Connection header received
0621 02:49:58.938109 (+ 2850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:58.938138 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:58.938710 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:58.939201 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:58.944039 (+ 4838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.945922 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.947337 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.948072 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.948197 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:58.952957 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:58.952965 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:58.952976 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:58.953004 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:58.956009 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:58.956721 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:58.956733 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:58.956739 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:58.956877 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:58.957242 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:58.957248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:58.957251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:58.957749 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:58.958060 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:58.958476 (+ 416us) server_negotiation.cc:299] Negotiation successful
0621 02:49:58.958939 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":415,"threads_started":1}
I20260621 02:49:58.960215 380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.945128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.62:36703 (local address 127.0.0.1:51396)
0621 02:49:58.945797 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.945838 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:58.946002 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:58.946441 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:58.946446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:58.947016 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:58.947316 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:58.947327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.948365 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.948373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:58.949491 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:58.949503 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.950641 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.950646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:58.950777 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:58.951478 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:58.951520 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:58.953836 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:58.956980 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:58.956991 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:58.956998 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:58.957388 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:58.957786 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:58.957792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:58.957796 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:58.957948 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:58.958468 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:58.958475 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:58.958646 (+ 171us) client_negotiation.cc:769] Sending connection context
0621 02:49:58.958857 (+ 211us) client_negotiation.cc:240] Negotiation successful
0621 02:49:58.959124 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":289,"threads_started":1}
I20260621 02:49:58.962095 376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } has no permanent_uuid. Determining permanent_uuid...
Jun 21 02:49:58 dist-test-slave-hvgn krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010198, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.60@KRBTEST.COM for kudu/127.9.166.61@KRBTEST.COM
Jun 21 02:49:58 dist-test-slave-hvgn krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.62@KRBTEST.COM
I20260621 02:49:58.976889 380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.962931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.61:39585 (local address 127.0.0.1:43310)
0621 02:49:58.963155 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.963175 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:58.963266 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:58.964103 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:58.964108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:58.964545 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:58.964760 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:58.964773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.965677 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.965683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:58.966855 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:58.966866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.967833 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.967838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:58.968038 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:58.968600 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:58.968623 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:58.970895 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:58.974279 (+ 3384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:58.974284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:58.974288 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:58.974825 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:58.975228 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:58.975233 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:58.975237 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:58.975435 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:58.975937 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:58.975942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:58.976088 (+ 146us) client_negotiation.cc:769] Sending connection context
0621 02:49:58.976231 (+ 143us) client_negotiation.cc:240] Negotiation successful
0621 02:49:58.976505 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260621 02:49:58.976915 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.963009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43310 (local address 127.9.166.61:39585)
0621 02:49:58.963186 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:58.963194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:58.963228 (+ 34us) server_negotiation.cc:407] Connection header received
0621 02:49:58.963420 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:58.963426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:58.963507 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:58.963606 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:58.964898 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.965544 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.966994 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.967686 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.967814 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:58.971092 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:58.971100 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:58.971104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:58.971153 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:58.974154 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:58.974939 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:58.974942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:58.974944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:58.975072 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:58.975548 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:58.975552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:58.975556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:58.975816 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:58.975958 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:58.976315 (+ 357us) server_negotiation.cc:299] Negotiation successful
0621 02:49:58.976525 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:49:58.977986 376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } has no permanent_uuid. Determining permanent_uuid...
I20260621 02:49:58.979480 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.963529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51410 (local address 127.9.166.62:36703)
0621 02:49:58.963726 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:58.963732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:58.963747 (+ 15us) server_negotiation.cc:407] Connection header received
0621 02:49:58.963796 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:58.963799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:58.963848 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:58.963949 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:58.965327 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.966384 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.967946 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.968631 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.968756 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:58.973359 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:58.973364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:58.973367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:58.973397 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:58.976430 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:58.977333 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:58.977338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:58.977341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:58.977498 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:58.977874 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:58.977881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:58.977883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:58.978232 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:58.978519 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:58.978878 (+ 359us) server_negotiation.cc:299] Negotiation successful
0621 02:49:58.979114 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Jun 21 02:49:58 dist-test-slave-hvgn krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010198, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.60@KRBTEST.COM for kudu/127.9.166.60@KRBTEST.COM
Jun 21 02:49:58 dist-test-slave-hvgn krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.166.61@KRBTEST.COM
I20260621 02:49:58.991889 379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.978929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54408 (local address 127.9.166.60:37189)
0621 02:49:58.979084 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:58.979089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:58.979105 (+ 16us) server_negotiation.cc:407] Connection header received
0621 02:49:58.979165 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:58.979171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:58.979241 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:58.979338 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:58.980129 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.981048 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.982372 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.982858 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.982950 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:58.986497 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:58.986502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:58.986505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:58.986531 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:58.989124 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:58.989922 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:58.989926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:58.989928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:58.990064 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:58.990481 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:58.990486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:58.990490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:58.990835 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:58.990987 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:58.991446 (+ 459us) server_negotiation.cc:299] Negotiation successful
0621 02:49:58.991607 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:49:58.992290 380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.978719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54408)
0621 02:49:58.978890 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:58.978908 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:58.978994 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:58.979370 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:58.979376 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:58.979878 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:58.979998 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:58.980006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.981175 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.981180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:58.982234 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:58.982246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.982996 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.983002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:58.983188 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:58.983888 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:58.983937 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:58.986307 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:58.989261 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:58.989266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:58.989270 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:58.989784 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:58.990202 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:58.990209 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:58.990216 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:58.990368 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:58.990973 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:58.990980 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:58.991142 (+ 162us) client_negotiation.cc:769] Sending connection context
0621 02:49:58.991321 (+ 179us) client_negotiation.cc:240] Negotiation successful
0621 02:49:58.991826 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":118}
I20260621 02:49:58.995898 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:58.980966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.9.166.61:39585)
0621 02:49:58.981173 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:58.981178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:58.981196 (+ 18us) server_negotiation.cc:407] Connection header received
0621 02:49:58.981243 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:58.981248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:58.981311 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:58.981416 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:58.982535 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.983492 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.984912 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:58.985416 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:58.985507 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:58.990291 (+ 4784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:58.990296 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:58.990299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:58.990327 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:58.993322 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:58.993977 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:58.993985 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:58.993988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:58.994137 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:58.994496 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:58.994501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:58.994505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:58.994839 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:58.995099 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:58.995312 (+ 213us) server_negotiation.cc:299] Negotiation successful
0621 02:49:58.995566 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:49:58.998409 376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507: Bootstrap starting.
I20260621 02:49:59.000870 9880 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260621 02:49:59.001876 376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:59.002889 376 log.cc:824] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:59.005988 376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507: No bootstrap required, opened a new log
I20260621 02:49:59.003206 32671 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "161c1bf37b924dfab17cac785abf5dab"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51410
I20260621 02:49:59.016713 376 raft_consensus.cc:348] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161c1bf37b924dfab17cac785abf5dab" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } } peers { permanent_uuid: "688b96cbba634b73a853dc26ca79dd0c" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } } peers { permanent_uuid: "c542b39816774df39653aa04b2f26507" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } }
I20260621 02:49:59.017108 376 raft_consensus.cc:729] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c542b39816774df39653aa04b2f26507, State: Initialized, Role: FOLLOWER
I20260621 02:49:59.017609 376 consensus_queue.cc:260] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161c1bf37b924dfab17cac785abf5dab" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } } peers { permanent_uuid: "688b96cbba634b73a853dc26ca79dd0c" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } } peers { permanent_uuid: "c542b39816774df39653aa04b2f26507" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } }
I20260621 02:49:59.018499 382 sys_catalog.cc:455] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161c1bf37b924dfab17cac785abf5dab" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } } peers { permanent_uuid: "688b96cbba634b73a853dc26ca79dd0c" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } } peers { permanent_uuid: "c542b39816774df39653aa04b2f26507" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } } }
I20260621 02:49:59.018854 382 sys_catalog.cc:458] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:49:59.019811 376 sys_catalog.cc:564] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507 [sys.catalog]: configured and running, proceeding with master startup.
W20260621 02:49:59.029495 393 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260621 02:49:59.029635 393 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260621 02:49:59.027500 309 master_runner.cc:186] Error getting master registration for 127.9.166.62:36703: OK, instance_id { permanent_uuid: "161c1bf37b924dfab17cac785abf5dab" instance_seqno: 1782010197929920 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 02:49:59.035367 309 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 02:49:59.070927 32696 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } attempt: 4
Jun 21 02:49:59 dist-test-slave-hvgn krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010197, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.62@KRBTEST.COM for kudu/127.9.166.60@KRBTEST.COM
I20260621 02:49:59.086841 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:59.071694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54414)
0621 02:49:59.071900 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:59.071920 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:59.072056 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:59.072514 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:59.072521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:59.072948 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:59.073121 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:59.073130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:59.074557 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:59.074562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:59.075576 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:59.075589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:59.076560 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:59.076566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:59.076789 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:59.077428 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:59.077446 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:59.079861 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:59.084005 (+ 4144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:59.084010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:59.084015 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:59.084507 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:59.084901 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:59.084906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:59.084909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:59.085079 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:59.085768 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:59.085773 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:59.085961 (+ 188us) client_negotiation.cc:769] Sending connection context
0621 02:49:59.086127 (+ 166us) client_negotiation.cc:240] Negotiation successful
0621 02:49:59.086429 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260621 02:49:59.086846 379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:59.071807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54414 (local address 127.9.166.60:37189)
0621 02:49:59.071989 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:59.071996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:59.072017 (+ 21us) server_negotiation.cc:407] Connection header received
0621 02:49:59.072285 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:59.072291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:59.072375 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:59.072476 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:59.073270 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:59.074416 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:59.075718 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:59.076420 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:59.076551 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:59.080056 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:59.080062 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:59.080067 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:59.080106 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:59.083859 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:59.084619 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:59.084624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:59.084629 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:59.084765 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:59.085199 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:59.085222 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:59.085226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:59.085636 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:59.085788 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:59.086251 (+ 463us) server_negotiation.cc:299] Negotiation successful
0621 02:49:59.086474 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260621 02:49:59.092764 32696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab: Bootstrap starting.
I20260621 02:49:59.096776 32696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:59.098073 32696 log.cc:824] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:59.101006 32696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab: No bootstrap required, opened a new log
I20260621 02:49:59.106040 32696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161c1bf37b924dfab17cac785abf5dab" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } } peers { permanent_uuid: "688b96cbba634b73a853dc26ca79dd0c" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } } peers { permanent_uuid: "c542b39816774df39653aa04b2f26507" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } }
I20260621 02:49:59.106391 32696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 161c1bf37b924dfab17cac785abf5dab, State: Initialized, Role: FOLLOWER
I20260621 02:49:59.106922 32696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161c1bf37b924dfab17cac785abf5dab" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } } peers { permanent_uuid: "688b96cbba634b73a853dc26ca79dd0c" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } } peers { permanent_uuid: "c542b39816774df39653aa04b2f26507" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } }
I20260621 02:49:59.107869 396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161c1bf37b924dfab17cac785abf5dab" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } } peers { permanent_uuid: "688b96cbba634b73a853dc26ca79dd0c" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } } peers { permanent_uuid: "c542b39816774df39653aa04b2f26507" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } } }
I20260621 02:49:59.108280 396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:49:59.109050 32696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:59.109143 32671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260621 02:49:59.109323 32671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:59.113420 32671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161c1bf37b924dfab17cac785abf5dab" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } } peers { permanent_uuid: "688b96cbba634b73a853dc26ca79dd0c" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } } peers { permanent_uuid: "c542b39816774df39653aa04b2f26507" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } }
I20260621 02:49:59.116534 32671 leader_election.cc:288] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [CANDIDATE]: Term 1 election: Requested vote from peers 688b96cbba634b73a853dc26ca79dd0c (127.9.166.61:39585), c542b39816774df39653aa04b2f26507 (127.9.166.60:37189)
I20260621 02:49:59.117384 351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "161c1bf37b924dfab17cac785abf5dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c542b39816774df39653aa04b2f26507"
I20260621 02:49:59.117858 351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507 [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:59.121304 351 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 161c1bf37b924dfab17cac785abf5dab in term 1.
I20260621 02:49:59.121999 32648 leader_election.cc:302] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 161c1bf37b924dfab17cac785abf5dab, c542b39816774df39653aa04b2f26507; no voters:
I20260621 02:49:59.122429 396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [term 1 FOLLOWER]: Leader election won for term 1
I20260621 02:49:59.124536 396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [term 1 LEADER]: Becoming Leader. State: Replica: 161c1bf37b924dfab17cac785abf5dab, State: Running, Role: LEADER
I20260621 02:49:59.125082 396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161c1bf37b924dfab17cac785abf5dab" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } } peers { permanent_uuid: "688b96cbba634b73a853dc26ca79dd0c" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } } peers { permanent_uuid: "c542b39816774df39653aa04b2f26507" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } }
I20260621 02:49:59.142168 404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 161c1bf37b924dfab17cac785abf5dab. Latest consensus state: current_term: 1 leader_uuid: "161c1bf37b924dfab17cac785abf5dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161c1bf37b924dfab17cac785abf5dab" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } } peers { permanent_uuid: "688b96cbba634b73a853dc26ca79dd0c" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } } peers { permanent_uuid: "c542b39816774df39653aa04b2f26507" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } } }
I20260621 02:49:59.142736 404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:59.144454 302 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } attempt: 5
Jun 21 02:49:59 dist-test-slave-hvgn krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1782010198, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.166.61@KRBTEST.COM for kudu/127.9.166.60@KRBTEST.COM
W20260621 02:49:59.159931 330 tablet.cc:2369] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:49:59.164280 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:59.145134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.166.60:37189 (local address 127.0.0.1:54420)
0621 02:49:59.145529 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0621 02:49:59.145548 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0621 02:49:59.145668 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0621 02:49:59.146311 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0621 02:49:59.146317 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0621 02:49:59.146888 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0621 02:49:59.147073 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:59.147087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:59.148647 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:59.148654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:59.150088 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0621 02:49:59.150101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0621 02:49:59.151237 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0621 02:49:59.151245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0621 02:49:59.151531 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:59.152264 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0621 02:49:59.152293 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0621 02:49:59.155344 (+ 3051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0621 02:49:59.161015 (+ 5671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:59.161021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:59.161027 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:59.161605 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:59.162120 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0621 02:49:59.162127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0621 02:49:59.162134 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0621 02:49:59.162338 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0621 02:49:59.163102 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0621 02:49:59.163108 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0621 02:49:59.163281 (+ 173us) client_negotiation.cc:769] Sending connection context
0621 02:49:59.163463 (+ 182us) client_negotiation.cc:240] Negotiation successful
0621 02:49:59.163799 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267}
I20260621 02:49:59.165405 379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:59.145626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.9.166.60:37189)
0621 02:49:59.145812 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:59.145818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:59.145835 (+ 17us) server_negotiation.cc:407] Connection header received
0621 02:49:59.145903 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:59.145910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:59.145993 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:59.146113 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:59.147268 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:59.148445 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:59.150250 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:59.150934 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:59.151043 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:59.155582 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:59.155588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:59.155606 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:59.155653 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:59.160859 (+ 5206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:59.161767 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:59.161776 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:59.161782 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:59.161970 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:59.162481 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:59.162488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:59.162493 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:59.162879 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:59.163030 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:59.164730 (+ 1700us) server_negotiation.cc:299] Negotiation successful
0621 02:49:59.164982 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260621 02:49:59.171687 32650 tablet.cc:2369] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260621 02:49:59.173310 302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 688b96cbba634b73a853dc26ca79dd0c: Bootstrap starting.
I20260621 02:49:59.177953 302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 688b96cbba634b73a853dc26ca79dd0c: Neither blocks nor log segments found. Creating new log.
I20260621 02:49:59.181829 302 log.cc:824] T 00000000000000000000000000000000 P 688b96cbba634b73a853dc26ca79dd0c: Log is configured to *not* fsync() on all Append() calls
I20260621 02:49:59.184851 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:59.139639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.9.166.61:39585)
0621 02:49:59.139852 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:59.139859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:59.146221 (+ 6362us) server_negotiation.cc:407] Connection header received
0621 02:49:59.146594 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:59.146599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:59.146689 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:59.146754 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:59.150102 (+ 3348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:59.151152 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:59.159663 (+ 8511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:59.160291 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:59.160386 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:59.172114 (+ 11728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:59.172125 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:59.172130 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:59.172167 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:59.176180 (+ 4013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:59.179315 (+ 3135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:59.179323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:59.179329 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:59.179538 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:59.181194 (+ 1656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:59.181199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:59.181203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:59.181609 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:59.181862 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:59.184058 (+ 2196us) server_negotiation.cc:299] Negotiation successful
0621 02:49:59.184398 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260621 02:49:59.188271 302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 688b96cbba634b73a853dc26ca79dd0c: No bootstrap required, opened a new log
I20260621 02:49:59.205541 411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260621 02:49:59.219789 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:59.140402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51424 (local address 127.9.166.62:36703)
0621 02:49:59.140637 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:59.140646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:59.191029 (+ 50383us) server_negotiation.cc:407] Connection header received
0621 02:49:59.191454 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:59.191459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:59.191548 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:59.191629 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:59.194596 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:59.195732 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:59.199506 (+ 3774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:59.200336 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:59.200427 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:59.206324 (+ 5897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:59.206331 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:59.206335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:59.206376 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:59.212706 (+ 6330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:59.214267 (+ 1561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:59.214310 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:59.214324 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:59.214832 (+ 508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:59.215660 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:59.215721 (+ 61us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:59.215800 (+ 79us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:59.216577 (+ 777us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:59.217025 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:59.218330 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0621 02:49:59.218947 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260621 02:49:59.226805 411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260621 02:49:59.229025 302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 688b96cbba634b73a853dc26ca79dd0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161c1bf37b924dfab17cac785abf5dab" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } } peers { permanent_uuid: "688b96cbba634b73a853dc26ca79dd0c" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } } peers { permanent_uuid: "c542b39816774df39653aa04b2f26507" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } }
I20260621 02:49:59.229547 302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 688b96cbba634b73a853dc26ca79dd0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 688b96cbba634b73a853dc26ca79dd0c, State: Initialized, Role: FOLLOWER
I20260621 02:49:59.231892 379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:49:59.189853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54436 (local address 127.9.166.60:37189)
0621 02:49:59.190236 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:59.190245 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:59.195671 (+ 5426us) server_negotiation.cc:407] Connection header received
0621 02:49:59.195921 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:59.195929 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:59.196013 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:59.196088 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:59.200067 (+ 3979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:59.201109 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:59.206108 (+ 4999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:49:59.206801 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:49:59.206901 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:49:59.222045 (+ 15144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0621 02:49:59.222052 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0621 02:49:59.222057 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0621 02:49:59.222114 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0621 02:49:59.226011 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:59.227301 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:59.227332 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:59.227337 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:59.227605 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0621 02:49:59.228319 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0621 02:49:59.228323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0621 02:49:59.228353 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0621 02:49:59.228961 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0621 02:49:59.229456 (+ 495us) server_negotiation.cc:1035] Waiting for connection context
0621 02:49:59.230523 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0621 02:49:59.231039 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260621 02:49:59.233898 302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 688b96cbba634b73a853dc26ca79dd0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161c1bf37b924dfab17cac785abf5dab" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } } peers { permanent_uuid: "688b96cbba634b73a853dc26ca79dd0c" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } } peers { permanent_uuid: "c542b39816774df39653aa04b2f26507" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } }
I20260621 02:49:59.235126 419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688b96cbba634b73a853dc26ca79dd0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161c1bf37b924dfab17cac785abf5dab" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } } peers { permanent_uuid: "688b96cbba634b73a853dc26ca79dd0c" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } } peers { permanent_uuid: "c542b39816774df39653aa04b2f26507" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } } }
I20260621 02:49:59.235875 419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688b96cbba634b73a853dc26ca79dd0c [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:49:59.236457 302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 688b96cbba634b73a853dc26ca79dd0c [sys.catalog]: configured and running, proceeding with master startup.
I20260621 02:49:59.236479 32745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "161c1bf37b924dfab17cac785abf5dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "688b96cbba634b73a853dc26ca79dd0c"
I20260621 02:49:59.236939 32745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 688b96cbba634b73a853dc26ca79dd0c [term 0 FOLLOWER]: Advancing to term 1
I20260621 02:49:59.240897 32745 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 688b96cbba634b73a853dc26ca79dd0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 161c1bf37b924dfab17cac785abf5dab in term 1.
I20260621 02:49:59.244220 351 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507 [term 1 FOLLOWER]: Refusing update from remote peer 161c1bf37b924dfab17cac785abf5dab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:49:59.245402 404 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c542b39816774df39653aa04b2f26507" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:49:59.247110 32745 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 688b96cbba634b73a853dc26ca79dd0c [term 1 FOLLOWER]: Refusing update from remote peer 161c1bf37b924dfab17cac785abf5dab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260621 02:49:59.256783 404 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "688b96cbba634b73a853dc26ca79dd0c" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260621 02:49:59.249007 32629 master_runner.cc:186] Error getting master registration for 127.9.166.61:39585: OK, instance_id { permanent_uuid: "688b96cbba634b73a853dc26ca79dd0c" instance_seqno: 1782010198417937 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260621 02:49:59.276721 32629 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260621 02:49:59.283015 382 sys_catalog.cc:455] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 161c1bf37b924dfab17cac785abf5dab. Latest consensus state: current_term: 1 leader_uuid: "161c1bf37b924dfab17cac785abf5dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161c1bf37b924dfab17cac785abf5dab" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } } peers { permanent_uuid: "688b96cbba634b73a853dc26ca79dd0c" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } } peers { permanent_uuid: "c542b39816774df39653aa04b2f26507" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } } }
I20260621 02:49:59.283339 382 sys_catalog.cc:458] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:49:59.287559 419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688b96cbba634b73a853dc26ca79dd0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 161c1bf37b924dfab17cac785abf5dab. Latest consensus state: current_term: 1 leader_uuid: "161c1bf37b924dfab17cac785abf5dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161c1bf37b924dfab17cac785abf5dab" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } } peers { permanent_uuid: "688b96cbba634b73a853dc26ca79dd0c" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } } peers { permanent_uuid: "c542b39816774df39653aa04b2f26507" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } } }
I20260621 02:49:59.287868 419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688b96cbba634b73a853dc26ca79dd0c [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:49:59.290014 417 mvcc.cc:205] Tried to move back new op lower bound from 7299113776088535040 to 7299113775621906432. Current Snapshot: MvccSnapshot[applied={T|T < 7299113776088535040}]
I20260621 02:49:59.296412 419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688b96cbba634b73a853dc26ca79dd0c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "161c1bf37b924dfab17cac785abf5dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161c1bf37b924dfab17cac785abf5dab" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } } peers { permanent_uuid: "688b96cbba634b73a853dc26ca79dd0c" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } } peers { permanent_uuid: "c542b39816774df39653aa04b2f26507" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } } }
I20260621 02:49:59.296743 419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688b96cbba634b73a853dc26ca79dd0c [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:49:59.298179 382 sys_catalog.cc:455] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "161c1bf37b924dfab17cac785abf5dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161c1bf37b924dfab17cac785abf5dab" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } } peers { permanent_uuid: "688b96cbba634b73a853dc26ca79dd0c" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } } peers { permanent_uuid: "c542b39816774df39653aa04b2f26507" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } } }
I20260621 02:49:59.298434 382 sys_catalog.cc:458] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507 [sys.catalog]: This master's current role is: FOLLOWER
I20260621 02:49:59.302379 411 catalog_manager.cc:1347] Generated new cluster ID: 975172ff82654a97818b2a1421fcc3f5
I20260621 02:49:59.302479 411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260621 02:49:59.317418 404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "161c1bf37b924dfab17cac785abf5dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161c1bf37b924dfab17cac785abf5dab" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } } peers { permanent_uuid: "688b96cbba634b73a853dc26ca79dd0c" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } } peers { permanent_uuid: "c542b39816774df39653aa04b2f26507" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } } }
I20260621 02:49:59.317739 404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:59.318080 404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "161c1bf37b924dfab17cac785abf5dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "161c1bf37b924dfab17cac785abf5dab" member_type: VOTER last_known_addr { host: "127.9.166.62" port: 36703 } } peers { permanent_uuid: "688b96cbba634b73a853dc26ca79dd0c" member_type: VOTER last_known_addr { host: "127.9.166.61" port: 39585 } } peers { permanent_uuid: "c542b39816774df39653aa04b2f26507" member_type: VOTER last_known_addr { host: "127.9.166.60" port: 37189 } } }
I20260621 02:49:59.318320 404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab [sys.catalog]: This master's current role is: LEADER
I20260621 02:49:59.339517 411 catalog_manager.cc:1370] Generated new certificate authority record
I20260621 02:49:59.341426 411 catalog_manager.cc:1504] Loading token signing keys...
I20260621 02:49:59.348345 438 catalog_manager.cc:1259] Loaded cluster ID: 975172ff82654a97818b2a1421fcc3f5
I20260621 02:49:59.348439 438 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 688b96cbba634b73a853dc26ca79dd0c: loading cluster ID for follower catalog manager: success
I20260621 02:49:59.351960 438 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 688b96cbba634b73a853dc26ca79dd0c: acquiring CA information for follower catalog manager: success
W20260621 02:49:59.356545 438 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260621 02:49:59.372957 411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 161c1bf37b924dfab17cac785abf5dab: Generated new TSK 0
I20260621 02:49:59.374306 411 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260621 02:49:59.495941 306 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:59.484615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.9.166.61:39585)
0621 02:49:59.484864 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:59.484874 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:59.493559 (+ 8685us) server_negotiation.cc:407] Connection header received
0621 02:49:59.494421 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:59.494427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:59.494510 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:59.494588 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:59.495689 (+ 1101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43320: BlockingRecv error: recv got EOF from 127.0.0.1:43320 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260621 02:49:59.500838 32698 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:59.492867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51432 (local address 127.9.166.62:36703)
0621 02:49:59.493113 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:59.493121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:59.497012 (+ 3891us) server_negotiation.cc:407] Connection header received
0621 02:49:59.497182 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:59.497191 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:59.497293 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:59.497420 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:59.500495 (+ 3075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51432: BlockingRecv error: recv got EOF from 127.0.0.1:51432 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260621 02:49:59.506390 379 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:59.503779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54438 (local address 127.9.166.60:37189)
0621 02:49:59.503956 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:59.503962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:59.504596 (+ 634us) server_negotiation.cc:407] Connection header received
0621 02:49:59.504936 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:59.504942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:59.505024 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:59.505095 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:59.506149 (+ 1054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54438: BlockingRecv error: recv got EOF from 127.0.0.1:54438 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260621 02:49:59.550189 306 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:59.534208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43332 (local address 127.9.166.61:39585)
0621 02:49:59.534407 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:59.534413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:59.547975 (+ 13562us) server_negotiation.cc:407] Connection header received
0621 02:49:59.548199 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:59.548206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:59.548385 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:59.548519 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:59.549826 (+ 1307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43332: BlockingRecv error: recv got EOF from 127.0.0.1:43332 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260621 02:49:59.598222 306 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:59.586143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43348 (local address 127.9.166.61:39585)
0621 02:49:59.586348 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:59.586356 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:59.596172 (+ 9816us) server_negotiation.cc:407] Connection header received
0621 02:49:59.596700 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:59.596707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:59.596807 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:59.596889 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:59.597968 (+ 1079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43348: BlockingRecv error: recv got EOF from 127.0.0.1:43348 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260621 02:49:59.646509 306 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:59.633769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43350 (local address 127.9.166.61:39585)
0621 02:49:59.633944 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:59.633950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:59.644536 (+ 10586us) server_negotiation.cc:407] Connection header received
0621 02:49:59.645012 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:59.645019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:59.645162 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:59.645239 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:59.646246 (+ 1007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43350: BlockingRecv error: recv got EOF from 127.0.0.1:43350 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260621 02:49:59.657028 306 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:59.654786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43366 (local address 127.9.166.61:39585)
0621 02:49:59.655057 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:59.655066 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:59.655648 (+ 582us) server_negotiation.cc:407] Connection header received
0621 02:49:59.655872 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:59.655878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:59.655975 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:59.656084 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:59.656767 (+ 683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43366: BlockingRecv error: recv got EOF from 127.0.0.1:43366 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
W20260621 02:49:59.670691 306 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:59.668513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.9.166.61:39585)
0621 02:49:59.668771 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:59.668780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:59.669294 (+ 514us) server_negotiation.cc:407] Connection header received
0621 02:49:59.669511 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:59.669515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:59.669587 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:59.669701 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:59.670451 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43382: BlockingRecv error: recv got EOF from 127.0.0.1:43382 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260621 02:49:59.692230 306 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:59.689942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43386 (local address 127.9.166.61:39585)
0621 02:49:59.690233 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:59.690243 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:59.690735 (+ 492us) server_negotiation.cc:407] Connection header received
0621 02:49:59.690970 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:59.690982 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:59.691071 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:59.691184 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:59.691973 (+ 789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43386: BlockingRecv error: recv got EOF from 127.0.0.1:43386 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260621 02:49:59.735757 306 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:59.733405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43390 (local address 127.9.166.61:39585)
0621 02:49:59.733655 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:59.733665 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:59.734264 (+ 599us) server_negotiation.cc:407] Connection header received
0621 02:49:59.734498 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:59.734502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:59.734578 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:59.734687 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:59.735489 (+ 802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43390: BlockingRecv error: recv got EOF from 127.0.0.1:43390 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260621 02:49:59.807574 306 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:59.805348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.9.166.61:39585)
0621 02:49:59.805623 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:59.805633 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:59.806146 (+ 513us) server_negotiation.cc:407] Connection header received
0621 02:49:59.806380 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:59.806391 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:59.806469 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:59.806579 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:59.807326 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43406: BlockingRecv error: recv got EOF from 127.0.0.1:43406 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260621 02:49:59.959224 306 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:49:59.956315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.9.166.61:39585)
0621 02:49:59.956492 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0621 02:49:59.956499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:49:59.957693 (+ 1194us) server_negotiation.cc:407] Connection header received
0621 02:49:59.957855 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:49:59.957864 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:49:59.957965 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:49:59.958086 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:49:59.958932 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43422: BlockingRecv error: recv got EOF from 127.0.0.1:43422 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260621 02:50:00.031632 393 catalog_manager.cc:1259] Loaded cluster ID: 975172ff82654a97818b2a1421fcc3f5
I20260621 02:50:00.031740 393 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507: loading cluster ID for follower catalog manager: success
I20260621 02:50:00.034119 393 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507: acquiring CA information for follower catalog manager: success
I20260621 02:50:00.035954 393 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c542b39816774df39653aa04b2f26507: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260621 02:50:00.222734 306 negotiation.cc:337] Failed RPC negotiation. Trace:
0621 02:50:00.220945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.9.166.61:39585)
0621 02:50:00.221155 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:00.221160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:00.221456 (+ 296us) server_negotiation.cc:407] Connection header received
0621 02:50:00.221623 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:00.221627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:00.221712 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:00.221810 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0621 02:50:00.222533 (+ 723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43430: BlockingRecv error: recv got EOF from 127.0.0.1:43430 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260621 02:50:00.359200 438 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 688b96cbba634b73a853dc26ca79dd0c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260621 02:50:00.745198 505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:00.739177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43446 (local address 127.9.166.61:39585)
0621 02:50:00.739687 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:00.739691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:00.740022 (+ 331us) server_negotiation.cc:407] Connection header received
0621 02:50:00.740192 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:00.740196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:00.740351 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:00.740460 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:50:00.740867 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:00.741587 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:00.742973 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:00.743428 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:00.743537 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:00.744056 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:50:00.744625 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:50:00.744751 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:00.744884 (+ 133us) server_negotiation.cc:299] Negotiation successful
0621 02:50:00.744957 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":295,"threads_started":1}
I20260621 02:50:00.757380 511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:00.751551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.9.166.60:37189)
0621 02:50:00.752058 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:00.752067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:00.752140 (+ 73us) server_negotiation.cc:407] Connection header received
0621 02:50:00.752405 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:00.752409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:00.752556 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:00.752743 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:50:00.753112 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:00.753816 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:00.755136 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:00.755586 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:00.755726 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:00.756234 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:50:00.756784 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:50:00.756955 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:00.757007 (+ 52us) server_negotiation.cc:299] Negotiation successful
0621 02:50:00.757090 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":239,"threads_started":1}
I20260621 02:50:00.770586 505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:00.763571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43454 (local address 127.9.166.61:39585)
0621 02:50:00.763730 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:00.763737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:00.764332 (+ 595us) server_negotiation.cc:407] Connection header received
0621 02:50:00.764632 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:00.764641 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:00.764761 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:00.764834 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:50:00.765715 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:00.766822 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:00.768654 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:00.769050 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:00.769177 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:00.769782 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:50:00.769909 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:50:00.770004 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:00.770159 (+ 155us) server_negotiation.cc:299] Negotiation successful
0621 02:50:00.770247 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260621 02:50:00.772033 511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:00.764437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54456 (local address 127.9.166.60:37189)
0621 02:50:00.764602 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:00.764607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:00.765617 (+ 1010us) server_negotiation.cc:407] Connection header received
0621 02:50:00.765778 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:00.765783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:00.765852 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:00.765945 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:50:00.766430 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:00.767270 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:00.769634 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:00.770275 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:00.770389 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:00.771289 (+ 900us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:50:00.771432 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:50:00.771520 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:00.771681 (+ 161us) server_negotiation.cc:299] Negotiation successful
0621 02:50:00.771756 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260621 02:50:00.772506 518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0621 02:50:00.763132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.9.166.62:36703)
0621 02:50:00.763697 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0621 02:50:00.763705 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0621 02:50:00.764284 (+ 579us) server_negotiation.cc:407] Connection header received
0621 02:50:00.764748 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0621 02:50:00.764757 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0621 02:50:00.765023 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0621 02:50:00.765203 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0621 02:50:00.766041 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:00.767060 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:00.769548 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0621 02:50:00.770084 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0621 02:50:00.770208 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0621 02:50:00.770947 (+ 739us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0621 02:50:00.771422 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0621 02:50:00.771533 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0621 02:50:00.772037 (+ 504us) server_negotiation.cc:299] Negotiation successful
0621 02:50:00.772128 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":488,"threads_started":1}
I20260621 02:50:00.777246 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 32629
I20260621 02:50:00.778273 32691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:00.891736 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 32629
I20260621 02:50:00.912810 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 32703
I20260621 02:50:00.913928 32765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:01.025272 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 32703
I20260621 02:50:01.045380 9880 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2e3lzq/build/asan/bin/kudu pid 309
I20260621 02:50:01.046568 371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260621 02:50:01.153016 9880 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2e3lzq/build/asan/bin/kudu with pid 309
2026-06-21T02:50:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5011 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9831 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (356767 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST